Contact Us for a Custom Demo or Trial: The best way to get started is to let us know your research goals and team needs. Request a customized demo to see how we can help.
Assay transform scripts can be configured to run when data is imported, updated, or both. (docs)
Study participants can be deleted from all datasets at once. (docs)
Names of data structures, including Sample Types, Source Types, and Assay Designs, may not contain certain special characters or substrings used internally. (docs)
Administrators can use a new table to locate field names with special characters on the server. While special characters are permitted in field names, they are not recommended. (docs) Also available in 24.11.9.
The wiki and announcements renderer for Markdown has been replaced. The new renderer is highly compatible with the previous renderer and adds several new features. (docs)
Set a timeout for read-only HTTP requests, after which long running processes and database queries will be killed. (docs)
When the server is configured to use only HTTPS, a Strict-Transport-Security header will be set to prevent future attempts to access the server over HTTP. (docs) Also available in 24.11.4.
Administrators can register a list of acceptable file extensions for uploads; others will be denied. If no list is provided, any extension is allowed. (docs)
Encoding of WebDav URLs has been made more consistent; users who have been relying on the previous behavior may need to make changes. (docs)
A strong Content Security Policy (CSP) running in enforce mode is highly recommended for all servers. Administrators can now customize elements of the CSP through the Admin Console. (docs)
New options to assist with troubleshooting Postgres database issues have been added to the Admin Console. (docs)
New documentation example of using an icon on a custom grid button. (docs)
Distribution Changes and Upgrade Notes
A strong Content Security Policy (CSP) is highly recommended for all servers. (docs)
Our recommended strong CSP blocks all inline script and prevents browsers from accessing unvetted external resources. This means the browser blocks all cross-site script (XSS) injection and cross-origin resource sharing attacks.
All LabKey Cloud deployments are now configured with a strong CSP in enforce mode.
All on-premise distributions are now configured by default with a strong CSP in report-only mode. We recommend switching this to enforce mode as soon as you have tested in report mode and are comfortable with the findings.
Important: If you are running on-premise and previously customized the CSP and are running it in enforce mode, you can continue to do so. If you have customized the CSP to run in report mode, we recommend removing the customization from your application.properties so that our default will be used instead. Once you are satisfied with the report results, you'll be able to move to enforce mode.
Users with on-premise installations who have already upgraded to use embedded Tomcat should follow these upgrade instructions. (upgrade on linux | upgrade on windows)
Users with on-premise installations that have not already migrated their server to use embedded Tomcat will need to follow an additional set of steps covered in the documentation archives. (migration docs)
The default configuration for Tomcat error pages has been improved to reduce unneeded information like the version number and stack trace.
Error logging has been improved for situations like port conflicts. (docs)
Support for the "Remote Login API" has been removed.
The "Vaccine Study Protocol" interface has been removed.
Support for FreezerPro integration has been removed.
Support for SampleMinded integration has been removed.
Support for additional date, time, and datetime parsing patterns has been deprecated.
"Assay Progress Reports" in studies have been deprecated.
Support for SQL Server 2016 has been deprecated and will be removed in a future release. (supported versions)
Support for object-level discussions has been deprecated.
Support for Ancillary Studies has been deprecated.
The Publish Study feature provides a more extensive mechanism for creating child studies. (docs)
Support for study protocol design tools has been deprecated.
The "Advanced import options" of choosing objects during import of folder archives, as well as having those imports applied to multiple folders simultaneously, have both been deprecated.
Client APIs and Development Notes
Rlabkey version 3.4.1 has been released, supporting the consistency improvements in webdav URLs. Note that this means version 3.4.1 requires LabKey version 24.12 or higher; earlier versions of LabKey must use Rlabkey version 3.4.0 or lower. (docs)
A new validator will identify external images referenced from Electronic Lab Notebooks. (docs)
A custom utility that will scan for strict CSP violations in wikis, participant views, and study descriptions is now available to LabKey Cloud and Premium Edition subscribers. (docs) Also available in 25.3.9.
Premium Resource for EHR (Electronic Health Records): Documentation of validation trigger levels has been added to assist developers. (docs)
The "Site Administrators" and "Site Developers" groups are no longer created by default or used for assigning permissions. (docs)
Permissions for users may need to be adjusted if they previously relied on these groups. In particular, report permissions that were previously automatically granted to site administrators must now be assigned directly. (docs)
By default, the "assigned-to" list for an issue tracker is now the "Site Users" group, where it was previously the "Site Administrators" group. (docs)
Distribution Changes and Upgrade Notes
A strong Content Security Policy (CSP) is highly recommended for all servers. (docs)
Our recommended strong CSP blocks all inline script and prevents browsers from accessing unvetted external resources. This means the browser blocks all cross-site script (XSS) injection and cross-origin resource sharing attacks.
All LabKey Cloud deployments and on-premise distributions are configured with a strong CSP in enforce mode as of version 25.7.
Important: Administrators should be carefully reviewing CSP reports and addressing any issues as soon as possible. Once you are satisfied with the report results, you'll be able to move to enforce mode. (docs)
The "Advanced import options" of choosing objects during import of folder archives, as well as having those imports applied to multiple folders simultaneously, have both been deprecated.
Support for additional date, time, and datetime parsing patterns has been removed.
"Assay Progress Reports" in studies have been removed.
Support for object-level discussions has been removed.
Support for Ancillary Studies has been removed.
The Publish Study feature provides a more extensive mechanism for creating child studies. (docs)
Support for study protocol design tools has been removed.
The behavior of lookups with values that look like numbers has been improved. We will attempt to resolve by alternate key (if allowLookupByAlternateKey is true) and then by primary key.
Our documentation example of an assay transformation script in R has been updated to show how to enable proper encoding of strings. Use the quote=TRUE and qmethod="double" parameters for the write.table. Your code may require similar updates. (docs
The Data Grid Tutorial is a quick hands-on demonstration you can try without any setup or registration. It shows you just a few of the ways that LabKey Server can help you:
Securely share your data with colleagues through interactive grid views
Collaboratively build and explore interactive visualizations
Drill down into de-identified data for study participants
Combine related datasets using data integration tools
To get started using LabKey products, you can contact us to tell us more about your research and goals. You can request a customized demo so we can help understand how best to meet your needs. In some cases we will encourage you to evaluate and explore with your own data using a custom trial instance. Options include:
LabKey Server Trial
LabKey Server Trial instances contain a core subset of features, and sample content to help get you started. Upload your own data, try tutorials, and even create a custom site tailored to your research and share it with colleagues. Your trial lasts 30 days and we're ready to help you with next steps toward incorporating LabKey Server into your research projects.Start here: Explore LabKey Server with a trial in LabKey Cloud
Try the core features of LabKey Biologics LIMS using our example data and tutorial walkthroughs. Your trial lasts 30 days and we're ready to help you with next steps toward incorporating LabKey Biologics into your work.Start here: Explore LabKey Biologics with a Trial
Explore LabKey Server with a trial in LabKey Cloud
To get started using LabKey Server and understanding the core functionality, contact us about your research needs and goals. Upon request we will set up a LabKey Cloud-based trial of LabKey Server for you.
You'll receive an email with details about getting started and logging in.
Trial server instances contain a subset of features, and some basic content to get you started. Upload your own data, try tutorials, and even create a custom site tailored to your research and share it with colleagues. Your trial lasts 30 days and we're ready to help you with next steps toward incorporating LabKey Server into your research projects.
Tours & Tutorials
Step by step introductions to key functionality of LabKey Server.
This topic helps you get started understanding how LabKey Server works and how it can work for you. It is intended to be used alongside a LabKey Trial Server. You should have another browser window open on the home page of your Trial.
The project and folder hierarchy is like a directory tree and forms the basic organizing structure inside LabKey Server. Everything you create or configure in LabKey Server is located in some folder. Projects are the top level folders, with all the same behavior, plus some additional configuration options; they typically represent a separate team or research effort.The Home project is a special project. On your Trial server, it contains the main welcome banner. To return to the home project at any time, click the LabKey logo in the upper left corner.The project menu is on the left end of the menu bar and includes the display name of the current project.Hover over the project menu to see the available projects, and folders within them. Click any project or folder name to navigate there.Any project or folder with subfolders will show / buttons for expanding and contracting the list shown. If you are in a subfolder, there will be a clickable 'breadcrumb' trail at the top of the menu for quickly moving up the hierarchy. The menu will scroll when there are enough items, with the current location visible and expanded by default.The project menu always displays the name of the current project, even when you are in a folder or subfolder. A link with the Folder Name is shown near the top of page views like the following, offering easy one click return to the main page of the folder.For more about projects, folders, and navigation, see Project and Folder Basics.
Tabs
Using tabs within a folder can give you new "pages" of user interface to help organize content. For an example of tabs in action, see the Research Study within the Example Project.When your browser window is too narrow to display tabs arrayed across the screen, they will be collapsed into a pulldown menu showing the current tab name and a (chevron). Click the name of the tab on this menu to navigate to it.For more about adding and customizing tabs, see Use Tabs.
Web Parts
Web parts are user interface panels that can be shown on any folder page or tab. Each web part provides some type of interaction for users with underlying data or other content.There is a main "wide" column on the left and narrower column on the right. Each column supports a different set of web parts. By combining and reordering these web parts, an administrator can tailor the layout to the needs of the users.For a hands-on example to try right now, explore the Collaboration Workspace project on your Trial Server.To learn more, see Add Web Parts and Manage Web Parts. For a list of the types of web parts available in a full installation of LabKey Server, see the Web Part Inventory.
Header Menus
In the upper right, icon menus offer:
: Click to open a site-wide search box.
: Shown only to Admins: Administrative options available to users granted such access. See below.
username: Login and security options; help links to documentation.
Admin MenuThe "first user" of this trial site will always be an administrator and have access to the menu. If that user adds others, they may or may not have the same menu of options available, depending on permissions granted to them.
Site >: Settings that pertain to the entire site.
Admin Console: In this Trial edition of LabKey Server, some fields are not configurable and may be shown as read-only. See Admin Console for details about options available in the full installation of LabKey.
Site Users, Groups, Permissions: Site-level security settings.
Create Project: Creates a new project (top-level folder) on the server.
Folder >: Settings for the current folder.
Permissions: Security configuration for the current folder.
Management: General configuration for the current folder.
Project Users and Settings: General configuration for the current project.
Page Admin Mode: Used to change page layout and add or remove UI elements.
Manage Views, Lists, Assays: Configuration for common data containers.
Manage Hosted Server Account: Return to the site from which you launched this trial server.
Go To Module >: Home pages for the currently enabled modules.
Security Model
LabKey Server has a group and role-based security model. Whether an individual is authorized to see a resource or perform an action is checked dynamically based on the groups they belong to and roles (permissions) granted to them. Learn more here: Security. Try a walkthrough using your Trial Server here: Exploring LabKey Security
Tools for Working Together
Collaborating with teams within a single lab or around the world is made easier when you share resources and information in an online workspace.
Message Boards: Post announcements and carry on threaded discussions. Learn more here.
Issue Trackers: Track issues, bugs, or other workflow tasks (like assay requests) by customizing an issue tracker. LabKey uses issue trackers to manage development issues and client requests. Learn more here.
Wikis: Documents written in HTML, Markdown, Wiki syntax, or plain text; they can include images, links, and live content from data tables. You're reading a Wiki page right now. Learn more here.
File Repositories: Upload and selectively share files and spreadsheets of data; connect with custom import methods. You can see an example here. Learn more here.
To learn more and try these tools for yourself, navigate to the Example Project > Collaboration Workspace folder of your Trial Server in one browser window, and open the topic Exploring LabKey Collaboration in another.
Tools for Data Analysis
Biomedical research data comes in many forms, shapes, and sizes. LabKey integrates directly with many types of instruments and software systems and with customization can support any type of tabular data.
Uploading Data: From dragging and dropping single spreadsheets to connecting a data pipeline to an outside location for incoming data, your options are as varied as your data. Learn about the options here: Import Data.
LabKey Data Grids: LabKey data grids show a configurable view of the underlying database table. To learn more, try a quick online tour of data grids.
Interpreting Instrument Data: Using assay designs and pipeline protocols, you can direct LabKey Server to correctly interpret complex instrument data during import. Learn more here: Assay Data.
Statistics and Queries: Add summary statistics to your data grids and write SQL queries within the UI. Learn more here: Column Summary Statistics and SQL Queries.
Visualizations: Create easy charts and plots backed by live data. Learn more here: Visualizations.
Reporting: Generate reports and query snapshots of your data. Use R, JavaScript, and LabKey plotting APIs to present your data in many ways. Learn more here: Reports and Charts.
To tour some example content and try these tools, navigate to the Example Project > Laboratory Data folder of your Trial Server in one browser window, and open the topic Exploring Laboratory Data in another.
Tools for Research Studies
Study folders organize research data about participants over time. There are many different ways to configure and use LabKey studies. Learn more here: Studies.
Study Schedules and Navigation: Dashboards for seeing at a glance what work is completed and what is scheduled helps coordinators manage research data collection. Learn more here: Study Navigation.
Participant and Date Alignment: By aligning all of your data based on the participant and date information, you can integrate and compare otherwise disparate test results. Explore the breadth of data for a single study subject, or view trends across cohorts of similar subjects. Learn more here: Study User Guide.
Publishing and Sharing: Select who should see which parts of your study data and analysis. Choose how to publish results, including many options for obscuring and protecting PHI (protected health information) and aggregating results. Learn more here: Study Administrator Guide and Publish a Study: Protected Health Information / PHI.
To learn more and try these tools, navigate to the Example Project > Research Study folder of your Trial Server in one browser window, and open the topic Exploring LabKey Studies in another.
What's Next?
Explore the example content on your Trial Server using one of these walkthroughs.
LabKey tools for collaboration include message boards, task trackers, file sharing, and wikis. The default folder you create in LabKey is a "Collaboration" folder which gives you many basic tools for working and sharing data with colleagues.
This topic is intended to be used alongside a LabKey Trial Server. You should have another browser window open to view the Example Project > Collaboration Workspace folder on your trial server.
The "Collaboration Workspace" folder in the "Example Project" shows three web parts on its main dashboard. Web parts are user interface panels and can be customized in many ways.
1. Learn About LabKey Collaboration Folders: a panel of descriptive information (not part of a default Collaboration folder).
2. Messages: show conversations or announcements in a messages web part
3. Task Tracker: LabKey's issue tracker tools can be tailored to a variety of uses.
To help show some of the collaborative options, this project also includes a few sample users with different roles. You can see a message from the "team lead" and there is also a task called "Get Started" listed as a "Todo".
A message board is a basic tool for communication; LabKey Message Boards can be customized to many use cases: from announcements to developer support and discussion.
Notice the first message, "Hello World".
Click View Message or Respond below it.
Click Respond.
Enter any text you like in the Body field. If you also change the Title, your response will have a new title but the main message thread will retain the existing title.
Notice that you can select other options for how your body text is rendered. Options: Plain Text, HTML, Markdown, or Wiki syntax.
Notice you could attach a file if you like.
Click Submit.
You are now viewing the message thread. Notice links to edit or delete. Since you an administrator on this server, you can edit messages others wrote, which would not be true for most users.
You may have also received an email when you posted your response. By default, you are subscribed to any messages you create or comment on. Click unsubscribe to see how you would reset your preferences if you don't want to receive these emails.
Click the Collaboration Workspace link near the top of the page to return to the main folder dashboard. These links are shown any time you are viewing a page within a folder.
Notice that on the main folder dashboard, the message board does not show the text of your reply, but just the note "(1 response)".
Click New to create a new message.
You will see the same input fields as when replying; enter some text and click Submit.
When you return to the main folder dashboard, you will see your new message.
An administrator can control many display aspects of message boards, including the level of detail, order of messages, and even what "Messages" are called.
The (triangle) menu in the upper right corner includes several options for customizing.
New: Create a new message.
View List: See the message board in list format.
Admin: Change things like display order, what messages are called, security, and what options users have when adding messages.
Email: Control when and how email notifications are sent about messages on this board.
Customize: Select whether this web part shows the "full" information about the message, as shown in our example, or just a simple preview.
More details about using message boards can be found in this topic: Use Message Boards.
Task Tracker
LabKey provides flexible tools for tracking tasks with multiple steps done by various team members. Generically referred to as "issue trackers" the example project includes a simple "Task Tracker".The basic life cycle of any task or issue moves through three states:
Open: someone decides something needs to be done; various steps and reassignments can happen, including prioritization and reassignments
Resolved: someone does the thing
Closed: the orginal requestor confirms the solution is correct
Navigate to the Collaboration Workspace.
Scroll down to the Task Tracker and click the title of the task, Get Started to open the detail view.
The status, assignment, priority, and other information are shown here. You can see that the team leader opened this task and added the first step: assign this task to yourself.
Click Update. Used when you are not "resolving" the issue, merely changing assignment or adding extra information.
Select your username from the Assigned To pulldown.
You could also change other information and provide an optional comment about your update.
Click Save.
Note: You may receive an email when you do this. Email preferences are configurable. Learn more here.
The task is now assigned to you. You can see the sequence of changes growing below the current issue properties.
Click Collaboration Workspace to return to the main page.
Notice the task is assigned to you now.
Click New Task to create a new task.
Enter your choice of title, notice that the default status "Open" cannot be changed, but you can change the priority, and enter other information. Note that setting the priority field is required.
Assign the task to the "lab technician" user.
Click Save to open the new issue.
When you return to the task list on the Collaboration Workspace page, you will see it listed as issue 2.
To show how the resolution process works, use the fact that you are an administrator and can use impersonation to take on another user's identity. Learn more here.
Select (Your Username) > Impersonate > User.
Choose "lab_technician" and click Impersonate. You are now seeing the page as the lab technician would. For example, you no longer have access to administrator options on the messages web part.
Click the title of the new task you assigned to the lab technician to open it.
Click Resolve.
Notice that by default, the resolved task will be assigned to the original user who opened it - in this case you!
The default value for the Resolution field is "Fixed", but you can select other options if appropriate.
Enter a comment saying you've completed the task, then click Save.
Click Stop Impersonating.
Open the task again and close it as yourself.
Enter a few more tasks to have more data for viewing.
When finished, return to the Collaboration Workspace main page.
The Task Tracker grid offers many options.
Search the contents by ID number or search term using the search box.
Sort and filter the list of tasks using the header menu for any column. Learn more here.
Create custom grid views (ways to view tasks), such as "All assigned to me" or "All priority 1 issues for the current milestone". Use (Grid Views) > Customize Grid and add filters and sorts here. Learn more here.
You could also customize the grid to expose other columns if helpful, such as the name of the user who originally opened the issue. Learn more here.
The generic name for these tools is "issue trackers" and they can be customized to many purposes. LabKey uses one internally for bug tracking, and client portals use them to track client questions and work requests.Learn about creating your own issue tracker in this topic: Tutorial: Issue Tracking.
Web parts are panels of user interface that display content to your users. As an administrator, you can customize what is shown on the page by using page admin mode. To add a new web part to any folder page:
Select > Page Admin Mode. Note that if you do not see this option, make sure you are logged in as an administrator and not impersonating another user.
Notice that <Select Web Part> pulldown menus appear at the bottom of the page. There is a wider "main" column on the left and a narrow column on the right; each column supports a different set of web parts.
Note: If both pulldown menus are stacked on the right, make your browser slightly wider to show them on separate sides.
Select the type of web part you want to create on the desired side. For example, to create a main panel wiki like the welcome panel shown in the Collaboration Workspace folder, select Wiki on the left.
Click Add.
The new web part will be added at the bottom of the column. While you are in Page Admin Mode, you can reposition it on the page using the (triangle) menu in the web part header. Move up or down as desired.
Click Exit Admin Mode in the upper right.
If you later decide to remove a web part from a page, the underlying content is not deleted. The web part only represents the user interface.For more about web part types and functionality, see Add Web Parts.
Add a Wiki
Wiki documents provide an easy way to display content. They can contain any text or visual information and be formatted in HTML, Markdown, Wiki syntax, or plain text by using "Wiki" format but not including formatting syntax. To create our first wiki, we use a wiki web part.
Click Create a new wiki page. Note that if a page named "default" already exists in the folder, the new web part will display it. In this case, create a new one by selecting New from the (triangle) menu in the header of the web part.
The Name must be unique in the folder.
The Title is displayed at the top of the page.
Choosing a Parent page lets you organize many wikis into a hierarchy. The table of contents on the right of the page you are reading now shows many examples.
Enter the Body of the wiki. To change what formatting is used, use the Convert to... button in the upper right. A short guide to the formatting you select is shown at the bottom of the edit page.
Notice that you can attach files and elect whether to show them listed at the bottom of the wiki page.
Click Save & Close.
Scroll down to see your new wiki web part.
To reopen for editing, click the (pencil) icon.
More details about creating and using wikis can be found in this topic: Wikis.
Add a List
A list is a simple table of data. For example, you might store a list of labs you work with.
Select > Manage Lists.
You will see a number of preexisting lists related to the task tracker.
Click Create New List.
Name: Enter a short name (such as "labs"). It must be unique in the folder.
Review the list properties available; for this first example, leave them unchanged.
Click the Fields section header to open it.
Click Manually Define Fields.
In the Name box of the first field, enter "LabName" (no spaces) to create a key column for our list. Leave the data type "Text".
After doing this, you can set the Key Field Name in the blue panel. Select LabName from the dropdown (the field you just added).
Use the icon to expand the field and see the options available. Even more can be found by clicking Advanced Settings.
Click Add Field and enter the name of each additional column you want in your list. In this example, we have added an address and contact person, both text fields. Notice the "Primary Key" field is specifed in the field details for LabName; it cannot be deleted and you cannot change its data type.
Click Save to create the "labs" list. You will see it in the grid of "Available Lists".
To populate this new empty list:
Click the list name ("labs") in the grid. There is no data to show.
Select (Insert data) > Insert new row.
Enter any values you like.
Click Submit.
Repeat the process to add a few more rows.
Click the Collaboration Workspace link to return to the main folder page.
To avoid overwriting this Example Project content with new tutorial content, you could create a new "Tutorials" project to work in. See Exploring Project Creation for a walkthrough.
Laboratory data can be organized and analyzed using LabKey Assay folders. This topic walks you through the basics using a simple Excel spreadsheet standing in for typically more complex instrument output.
This topic is intended to be used alongside a LabKey Trial Server. You should have another browser window open to view the Example Project > Laboratory Data folder.
The "Laboratory Data" folder is an Assay folder. You'll see four web parts in our example:
1. Learn About LabKey Assay Folders: a panel of descriptive information (not part of a default Assay folder).
2. Assay List: a list of assays defined in the folder. Here we have "Blood Test Data."
3. Blood Test Data Results: A query web part showing a grid of data.
4. Files: a file repository where you can browse files in this container or upload new ones.
The folder includes a simple assay design created from the "Standard Assay Type" representing some blood test results. A single run (spreadsheet) of data has been imported using it. Continue below to try out the assay tools.
Try It Now
This section helps you try some key features of working with laboratory data.
Assay List: Browse and use assay designs for data and metadata
Using an Assay List web part, you can browse existing assay designs and as an administrator, add new ones. An assay design tells the server how to interpret uploaded data. Click the name of the design in this web part to see the run(s) that have been uploaded.
Click the name Blood Test Data in the Assay List to see the "Runs" imported using it.
In this case, a single run (spreadsheet) "BloodTest_Run1.xls" has been imported. Click the Assay ID (in this case the file name) to see the results.
This example shows blood data for 3 participants over a few different dates. White blood counts (WBC), mean corpuscular volume (MCV), and hemoglobin (HGB) levels have been collected in an Excel spreadsheet.To see the upload process, we can simulate importing the existing run. We will cancel the import before any reimporting actually happens.
Click Re-Import Run above the grid.
The first page shows:
Assay Properties: These are fixed properties, like the assay design name, that are read only for all imports using this design.
Batch Properties: These properties will apply to a set of files uploaded together. For example, here they include specifying how the data is identified (the Participant/Visit setting) and selecting a target study to use if any data is to be copied. Select /Example Project/Research Study (Research Study).
Click Next.
On the next page, notice the batch properties are now read only.
Below them are Run Properties to specify; these could be entered differently for each run in the assay, such as the "Assay ID". If not provided, the name of the file is used, as in our example.
Here you also provide the data from the assay run, either by uploading a data file (or reusing the one we already uploaded) or entering your own new data.
Click Show Expected Data Fields. You will see the list of fields expected to be in the spreadsheet. When you design the assay, you specify the name, type, and other properties of expected columns.
Click Download Spreadsheet Template to generate a blank template spreadsheet. It will be named "data_<datetimestamp>.xls". You could open and populate it in Excel, then upload it to this server as a new run. Save this template file and use it when you get to the Files section below.
Since we are just reviewing the process right now, click Cancel after reviewing this page.
Understand Assay Designs
Next, review what the assay design itself looks like. You need to be an administrator to perform this step.
From the Blood Test Data Runs page, select Manage Assay Design > Edit assay design to open it.
The sections correspond to how a user will set properties as they import a run. Click each section heading to open it.
Assay Properties: Values that do not vary per run or batch.
Batch Fields: Values set once for each batch of runs.
Run Fields: Values set per run; in this case none are defined, but as you saw when importing the file, some built in things like "Assay ID" are defined per run.
Results Fields: This is the heart of the assay design and where you would identify what columns (fields) are expected in the spreadsheet, what their data type is and whether they are required.
Click the for a row to see or set other properties of each column. With this panel open, click Advanced Settings for even more settings and properties, including things like whether a value can be used as a "measure" in reporting, as shown here for the WBC field. Learn more about using the field editor in this topic: Field Editor.
Review the contents of this page, but make no changes and click Cancel.
Return to the main folder page by clicking the Laboratory Data link near the top of the page.
Blood Test Data Results
The assay results are displayed in a web part on the main page. Click the Laboratory Data link if you navigated away, then scroll down. Explore some general features of LabKey data grids using this web part:Column Header Menus: Each column header has a menu of options to apply to the grid based on the values in that column.
Sort:
Click the header of the WBC column and select Sort Descending.
Notice the (sort) icon that appears in the column header.
Filter:
Click the header of the Participant ID column and select Filter... to open a popup.
Use the checkboxes to "Choose Values". For this example, uncheck the box for "202".
The "Choose Values is only available when there are a limited number of distinct values.
Switching to the "Choose Filters" tab in this popup would let you use filter expressions instead.
Click OK to apply the filter.
Notice the (filter) icon shown in the column header. There is also a new entry in the filter panel above the grid - you can clear this filter by clicking the for the "Participant <> 202" filter.
Summary Statistics:
Click the header of the MCV column and select Summary Statistics... to open a popup.
Select the statistics you would like to show (the values are previewed in the popup.)
Premium Feature: Many summary statistics shown are only available with premium editions of LabKey Server. Learn more here.
In this case, Mean and Standard Deviation might be the most interesting. Click Apply.
Notice the new row at the bottom of the grid showing the statistics we selected for this column.
The view of the grid now includes a sort, a filter, and a summary statistic.Notice the message "This grid view has been modified" was added when you added the summary statistic. The grid you are now seeing is not the original default. You have not changed anything about the underlying imported data table, merely how you see it in this grid. Saving:
To save this changed grid view, making it persistent and sharable with others, follow these steps:
Click Save next to the grid modification message.
Select Named and give the new grid a name (such as "Grid with Statistics").
Check the box for "Make this grid view available to all users."
Click Save.
Notice that the grid now has Grid With Statistics shown above it.
To switch between grid views, use the (Grid Views) menu in the grid header. Switch between "default" and "Grid With Statistics" to see the difference.
Return to the "default" view before proceeding with this walkthrough.
Visualizations and Reports:There is not enough data in this small spreadsheet to make meaningful visualizations or reports, but you can see how the options work:
Column Visualizations: Each column header menu has a set of visualizations available that can be displayed directly in the grid view. Options vary based on the data type. For example, from the "WBC" column header menu, choose Box and Whisker.
Quick Charts: Choose Quick Chart from the header menu of any column to see a quick best guess visualization of the data from that column. The default is typically a box and whisker plot.
While similar to the column chart above, this chart creates a stand alone chart that is named and saved separately from the grid view.
It also opens in the "Chart Wizard", described next, which offers many customization options.
Note that a chart like this is backed by the "live" data in the source table. If you change the underlying data (either by editing or by adding additional rows to the table) the chart will also update automatically.
Plot Editor: More types of charts and many more configuration options are available using the common plot editor, the "Chart Wizard".
If you are already viewing a plot, you can change the plot type immediately by clicking Edit and then Chart Type.
If you navigated elsewhere, you can open the plot editor by returning to the grid view and selecting (Charts) > Create Chart above the grid.
Select the type of chart you want using the options along the left edge.
Learn more about creating and customizing each chart type in the documentation. You can experiment with this small data set or use the more complex data when Exploring LabKey Studies.
The Files web part lists the single assay run as well as the log file from when it was uploaded.To explore the abilities of the file browser, we'll create some new assay data to import. You can also click here to download ours if you want your data to match our screenshots without creating your own: data_run2.xls
Enter some values. Some participant ID values to use are 101, 202, 303, 404, 505, and 606, though any integer values will be accepted. You can ignore the VisitID column and enter any dates you like. The WBC and MCV columns are integers, HGB is a double. The more rows of data you enter, the more "new" results you will have available to "analyze."
Save the spreadsheet with the name "data_run2.xls" if you want to match our instructions.
Click here to download ours if you want your data to match our screenshots: data_run2.xls
Return to the Example Project > Laboratory Data folder.
Drag and drop the "data_run2.xls" file into the Files web part to upload it.
Select it using the checkbox and click Import Data.
Scroll down to find the "Import Text or Excel Assay" category.
Select Use Blood Test Data - the assay design we predefined. Notice you also have the option to create a new "Standard" assay design instead if you wanted to change the way it is imported.
Click Import.
You will see the Assay Properties and be able to enter Batch Properties; select "Example Project/Research Study" as the Target Study.
Click Next.
On the Run Properties and Data File page, notice your "data_run2.xls" file listed as the Run Data. You don't need to make any changes on this page.
Click Save and Finish.
Now you will see the runs grid, with your new run added to our original sample.
Click the file name to see the data you created.
Notice the (Grid Views) menu includes the custom grid view "Grid with Statistics" we created earlier - select it.
Notice the grid also shows a filter "Run = 2" (the specific run number may vary) because you clicked the run name to view only results from that run. Click the for the filter to delete it.
You now see the combined results from both runs.
If you saved any visualizations earlier using the original spreadsheet of data, view them now from the (Charts/Reports) menu, and notice they have been updated to include your additional data.
LabKey can help you create a new assay design from a spreadsheet of your own. Choose something with a few columns of different types, or simply add a few more columns to the "data_run2.xls" spreadsheet you used earlier. If you have existing instrument data that is in tabular format, you can also use that to complete this walkthrough.
Note that LabKey does have a few reserved names in the assay framework, including "container, created, createdBy, modified, and modifiedBy", so if your spreadsheet contains these columns you may encounter mapping errors if you try to create a new assay from it. There are ways to work around this issue, but for this getting-started tutorial, try renaming the columns or using another sample spreadsheet.
Here we use the name "mydata.xls" to represent your own spreadsheet of data.
Navigate to the main page of the Laboratory Data folder.
Drag and drop your "mydata.xls" file to the Files web part.
Select the file and click Import Data.
In the popup, choose "Create New Standard Assay Design" and click Import.
Give your new assay design a Name, such as "MyData".
Notice the default Location is the current project. Select instead the "Current Folder (Laboratory Data)".
The Columns for Assay Data section shows the columns imported from your spreadsheet and the data types the server has inferred from the contents of the first few rows of your spreadsheet.
If you see a column here that you do not want to import, simply uncheck it. If you want to edit column properties, you can do that using the (triangle) button next to the column name.
You can change the inferred data types as necessary. For example, if you have a column that happens to contain whole number values, the server will infer it is of type "Integer". If you want it to be "Number (Double)" instead, select that type after clicking the (caret) button next to the type. If a column happens to be empty in the first few rows, the server will guess "Text (String)" but you can change that as well.
Column Mapping below these inferrals is where you would map things like Participant ID and Date information. For instance, if you have a column called "When" that contains the date information, you can tell the server that here. It is not required that you provide mappings at all.
When you are satisfied with how the server will interpret your spreadsheet, scroll back up and click Begin Import.
You will be simultaneously creating the assay design (for this and future uses) and importing this single run. Enter batch properties if needed, then click Next.
Enter run properties if needed, including an Assay ID if you want to use a name other than your file name.
Click Show Expected Data Fields to review how your data will be structured. Notice that if you didn't provide mappings, new columns will be created for Specimen, Participant, Visit ID, and Date.
Click Save and Finish.
You now see your new run in a grid, and the new assay design has been created.
Click the filename (in the Assay ID column) to see your data.
Click column headers to sort, filter, or create quick visualizations.
Select Manage Assay Design > Edit assay design to review the assay design.
Click Laboratory Data to return to the main folder page.
Notice your new assay design is listed in the Assay List.
You can now upload and import additional spreadsheets with the same structure using it. When you import .xls files in the future, your own assay design will be one of the options for importing it.
Understand Link to Study
You may have noticed the Linked to Research Study column in the Blood Test Data Results web part. The sample assay design includes linking data automatically to the Example Project/Research Study folder on your Trial Server."Linking" data to a study does not copy or move the data. What is created is a dynamic link so that the assay data can be integrated with other data in the study about the same participants. When data changes in the original container, the "linked" set in the study is also updated.Click the View Link to Study History link in the Blood Test Data Results web part.You will see at least one link event from the original import when our sample data was loaded and linked during the startup of your server (shown as being created by the "team lead"). If you followed the above steps and uploaded a second "data_run2.xls" spreadsheet, that will also be listed, created and linked by you.
Click View Results to see the result rows.
Click one of the links in the Linked to Research Study column. You see what appears to be the same grid of data, but notice by checking the project menu or looking at the URL that you are now in the "Research Study" folder. You will also see the tabs present in a study folder.
Click the Clinical and Assay Data tab.
Under Assay Data you will see the "Blood Test Data" linked dataset. In the topic Exploring LabKey Studies we will explore how that data can now be connected to other participant information.
Using the project menu, return to the Laboratory Data folder.
More Tutorials
Other tutorials using "Assay" folders that you can run on your LabKey Trial Server:
To avoid overwriting this Example Project content with new tutorial content, you could create a new "Tutorials" project to work in. See Exploring Project Creation for a walkthrough.
Using LabKey Studies, you can integrate and manage data about participants over time. Cohort and observational studies are a typical use case. Learn more in this topic.
This topic is intended to be used alongside a LabKey Trial Server. You should have another browser window open to view the Example Project > Research Study folder.
The "Research Study" folder contains a simple fictional study. There are 5 tabs in the default LabKey study folder; the main landing page is the Overview tab where you will see three web parts:
1. Learn About LabKey Study Folders: a panel of descriptive information (not part of a default Study folder)
2. Study Data Tools: commonly used tools and settings.
3. Study Overview: displays study properties and quick links to study navigation and management options.
A study has five tabs by default. Each encapsulates functions within the study making it easier to find what you need. Click the tab name to navigate to it. Returning to the Research Study folder returns you to the Overview tab.Participants: View and filter the participant list by cohort; search for information about individuals.Clinical and Assay Data: Review data and visualizations available; create new joined grids, charts, charts and reports.The Data Views web part lists several datasets, joined views, and reports and charts.
Hover over a name to see a preview and some metadata about it.
Click a name or use the (Details) link to open an item.
The Access column indicates whether special permissions have been set for an item. Remember that only users granted "Read" access to this folder can see any content here, so seeing "public" means only that it is shared with all folder users. Other options are "private" (only visible to the creator) and custom (shared with specific individuals). See Configure Permissions for Reports & Views for more information.
Datasets:
Demographic datasets contain a single row per participant for the entire study.
Clinical datasets can contain many rows per participant, but only one per participant and date combination.
Assay datasets, typically results from instrument tests, can contain many rows per participant and date.
Manage: Only administrators can access this dashboard for managing the study.
Try It Now
This example study includes a simplified research scenario. Let's explore some key feature areas on each tab:
The Study Overview web part shows properties and introductory information about your study.
Click the Overview tab.
Click the (pencil) icon on the Study Overview web part.
Review and change study properties:
Label: By default, the folder name, here "Research Study," is used. You can edit to make it more descriptive, such as "HIV Study".
Investigator: Enter your own name to personalize this example.
Grant/Species: If you enter the grant name and/or species under study, they will be displayed. These fields also enable searches across many study folders to locate related research. Enter a grant name to see how it is shown.
Description/Render Type: Our example shows some simple HTML formatted text. You can include as much or as little information here as you like. Select a different Render Type to use Markdown, Wiki syntax, or just plain text.
Protocol Documents: Attach documents if you like - links to download will be included in the web part.
Timepoint Type: Studies can use several methods of tracking time; this decision is fixed at the time of study creation and cannot be modified here. See Visits and Dates to learn more.
Start/End Date: See and change the timeframe of your study if necessary. Note that participants can also have individual start dates. Changing the start date for a study in progress should be done with caution.
Subject Noun: By default, study subjects are called "participants" but you can change that here to "subject," "mouse," "yeast," or whatever noun you choose. Try changing these nouns to "Subject" and "Subjects".
Subject Column Name: Enter the name of the column in your datasets that contains IDs for your study subjects. You do not need to change this field to match the subject nouns you use.
When finished, click Submit and see how this information is displayed. Notice the "Participants" tab name is changed.
Reopen the editor and change the subject noun back to "Participant[s]" to restore the original tab and tool names for this walkthrough.
Track Overall Progress
Return to the Overview tab if you navigated away.
Click the Study Navigator link or the small graphic in the Study Overview web part.
The Study Navigator shows you at a glance how much data is available in your study and when it was collected.
Use the Participant's current cohort dropdown to see collection by cohort.
Use the checkboxes to switch between seeing counts of participants or rows or both.
Click the number at the intersection of any row and column to see the data. For example, Lab Results in month two look like this:
View Participant Data
Within a study you can dive deep for all the information about a single participant of interest.
Click the Participants tab.
If you know the participant ID, you can use the search box to find their information.
The Participant List can be quickly filtered using the checkboxes on the left.
Use the Filter box to narrow the list if you know part of the participant ID.
Hover over a label to see the group member IDs shown in bold. Click a label to select only that filter option in that category. Here we see there are 8 participants enrolled receiving ARV treatment.
Click any participant ID to see all the study data about that participant.
The details of any dataset can be expanded/collapsed using the and icons.
Click the Search For link above the report to search the entire site for other information about that participant. In this case, you will also see results from the "Laboratory Data" folder in this project.
Integrate Data Aligned by Participant and Date
Study datasets can be combined to give a broad picture of trends within groups over time.
Click the Clinical and Assay Data tab.
There are three primary datasets here: Demographics, Physical Exam, and Lab Results.
Click "Joined View: Physical Exam and Demographics" to open an integrated custom grid.
This grid includes columns from two datasets. To see how it was created:
Select (Grid Views) > Customize Grid.
Scroll down on the Available Fields side to Datasets and click the to expand it. Listed are the two other datasets.
Expand the "Demographics" node by clicking the .
Scroll down and notice the checked fields like Country and Group Assignment which appear in our joined view.
Scroll down on the Selected Fields side to see these fields shown.
You can use checkboxes to add more fields to what is shown in the grid, drag and drop to rearrange columns in your view, and use the and icons for selected fields to edit display titles or delete them from the view.
Click View Data when finished to see your changes.
Notice the message indicating you have unsaved changes. Click Revert to discard them.
Customize Visualizations and Reports
In Exploring Laboratory Data we show you how to create charts and plots based on single columns or tables. With the integration of diverse data in a study, you can easily create visualization and reports across many tables, backed by live data. Our example study includes a few examples.
On the Clinical and Assay Data tab, click Bar Chart: Blood Pressure by Cohort and Treatment.
Here we see plotted a measure from the "Physical Exam" dataset (Systolic Blood Pressure) against cohort and treatment group data from the "Demographics" dataset.
Click Edit in the upper right to open the plot for editing.
Click Chart Type to open the chart wizard.
Columns on the right can be dragged and dropped into the plot attribute boxes in the middle.
Select a different plot type using the options to the left. The plot editor will make a best effort to retain plot attribute column selections, though not all attributes apply to each chart type.
Click Box.
Notice that the X and Y axis selections are preserved.
Drag the column "Study: Cohort" to the "Color" box.
Drag the column "Gender" to the "Shape" box.
Click Apply to see the new box plot. At present, only the outliers make use of the color and shape selections.
Click Chart Layout.
Give the plot a new title, like "Systolic - Box Plot"
Change Show Points to All.
Check the box to Jitter Points; otherwise points will be shown in a single column.
Scroll down to see the controls for plot line and fill colors.
Choose a different Color Palette from which the point colors will be selected. Shown here, "Dark".
Click Apply to see the revised box plot.
Click Save As to save this as a new visualization and preserve the original bar chart.
Give the new report a name and click Save in the popup.
Manage Your Study
On the Manage tab, you can control many aspects of your study. For example, click Manage Cohorts to review how cohorts were assigned and what the other options are:
Assignment Mode: Cohorts can be simple (fixed) or advanced, meaning they can change during the study.
Assignment Type: You can manually assign participants to cohorts, or have assignments made automatically based on a dataset.
Automatic Participant/Cohort Assignment: Choose the dataset and column to use for assigning cohorts.
Optional
You can experiment by changing which column is used to define cohorts: For instance, choose "Country" and click Update Assignments.
Notice the new entries under Defined Cohorts and new assignments in the Participant-Cohort Assignments web part.
Click the Participants tab.
Now you see under Cohorts that both the original cohorts and new ones are listed. Using the hover behavior, notice that the original "Group 1 and Group 2" cohorts are now empty and participants can quickly be filtered by country.
Go back in your browser window or click the Manage tab and Manage Cohorts to return to the cohort page.
Under Defined Cohorts you can click Delete Unused, then return to the Participants tab to see they are gone.
Click the Clinical and Assay Data tab. Click Bar Chart: Blood Pressure by Cohort and Treatment and you will see it has been automatically updated to reflect the new cohort division by country.
Return the original cohorts before moving on:
On the Manage tab, click Manage Cohorts.
Restore the original assignments based on the Demographics/Group Assignment field.
Access to read and edit information in folders is generally controlled by the LabKey role based security model. Within a study, you gain the additional option of dataset level security.
On the Manage tab, click Manage Security.
Review the Study Security Type: Datasets can be read-only or editable, under either type of security:
Basic security: folder permissions determine access to all datasets
Custom security: folder permissions can be set by dataset for each group with folder access
Change the type to Custom security with editable datasets. Notice the warning message that this can change who can view and modify data.
Click Update Type.
When you change to using 'custom' security, two additional web parts are added to the page:
Study Security: Use radio buttons to grant access to groups. Click Update after changing. In the screenshot below, we're changing the "Study Team" and "Example Team Members" groups to "Per Dataset" permissions.
Per Dataset Permissions: Once any group is given per-dataset permissions using the radio buttons, you will have the option to individually set permission levels for each group for each dataset.
Click Save when finished, or to revert to the configuration before this step, set all for "Example Team Members" to read, and set all for "Study Team" to edit.
Learn About Protecting PHI
When displaying or exporting data, Protected Health Information (PHI) that could be used to identify an individual can be protected in several ways.Alternate Participant IDs and AliasesIf you want to share data without revealing participant IDs, you can use a system of alternates or aliases so that you can still show a consistent ID for any set of data, but it is not the actual participant ID.
On the Manage tab, click Manage Alternate Participant IDs and Aliases.
Alternate participant IDs allow you to use a consistent prefix and randomized set of the number of digits you choose.
Dates are also offset by a random amount for each participant. Visit-date information could potentially isolate the individual, so this option obscures that without losing the elapsed time between visits which might be relevant to your study.
Participant Aliases lets you specify a dataset containing specific aliases to use. For instance, you might use a shared set of aliases across all studies to connect related results without positively identifying individuals.
Mark Data as PHIThere are four levels to consider regarding protection of PHI, based on how much information a user will be authorized to see. For PHI protection to be enforced, you must BOTH mark data as PHI and implement viewing and export restrictions on your project. Data in columns can be marked as:
Restricted: The most sensitive information not shared with unauthorized users, even those authorized for Full PHI.
Full PHI: Only shared when users have permission to see all PHI. The user can see everything except restricted information.
Limited PHI: Some information that is less sensitive can be visible here.
On the Clinical and Assay Data tab, open the dataset of interest. Here we use the Demographics dataset.
Above the grid, click Manage.
Click Edit Definition.
Click the Fields section to open it.
Find the field of interest, here "Status of Infection."
Click the to expand it.
Click Advanced Settings.
In the popup, you will see the current/default PHI Level: "Not PHI".
Make another selection from the dropdown to change; in this example, we choose "Full PHI".
Click Apply.
Adjust PHI levels for as many fields as necessary.
Scroll down and click Save.
Click View Data to return to the dataset.
Note that these column markings are not sufficient to protect data from view to users with read access to the folder. The levels must be enforced with UI implementation, perhaps including users declaring their PHI level and agreeing to a custom terms of use. For more information on a LabKey implementation of this behavior, see Compliance Features.
To export or publish data at a given PHI level:
From the Manage tab, click Export Study.
On the export page, notice one of the Options is Include PHI Columns. Select what you want included in your export:
Restricted, Full, and Limited PHI (default): Include all columns.
Full and Limited PHI: Exclude only the Restricted PHI columns.
Limited PHI: Exclude the Full PHI and Restricted columns.
Uncheck the checkbox to exclude all PHI columns.
If you marked the demographics column as "Full PHI" above, select "Limited PHI"to exclude it. You can also simply uncheck the Include PHI Columns checkbox.
Click Export.
Examine the downloaded archive and observe that the file named "ARCHIVE_NAME.folder.zip/study/datasets/dataset5001.tsv" does not include the data you marked as PHI.
Publish the Study
Publishing a LabKey study allows you to select all or part of your results and create a new published version.
Click the Manage tab.
Click Publish Study (at the bottom).
The Publish Study Wizard will guide you through selecting what to publish.
By default, the new study will be called "New Study" and placed in a subdirectory of the current study folder.
Select the participants, datasets, timepoints, and other objects to include. On the Datasets step, you can elect to have the study refresh data if you like, either manually or nightly.
The last page of the publish wizard offers Publish Options including obscuring information that could identify individuals and opting for the level of PHI you want to publish.
Click Finish at the end of the wizard to create the new study folder with the selected information.
Explore the new study, now available on the projects menu.
More Tutorials
Other tutorials using "Study" folders that you can run on your LabKey Trial Server:
To avoid overwriting this Example Project content with new tutorial content, you could create a new "Tutorials" project to work in. See Exploring Project Creation for a walkthrough.
Learn more about how LabKey manages security through roles and groups in this topic.
This topic is intended to be used alongside a LabKey Trial Server. You should have another browser window open to view the Example Project folder. This walkthrough also assumes you are the original creator of the trial server and are an administrator there, giving you broad access site-wide.
Our Example Project contains three subfolders, intended for different groups of users:
Collaboration Workspace: The entire team communicates here and shares project-wide information.
Laboratory Data: A lab team performs tests and uploads data here, perhaps performing basic quality control.
Research Study: A team of researchers is exploring a hypothesis about HIV.
LabKey's security model is based on assignment of permission roles to users, typically in groups.
Project Groups
Groups at the project level allow you to subdivide your project team into functional subgroups and grant permissions on resources to the group as a whole in each folder and subfolder. While it is possible to assign permissions individually to each user, it can become unwieldy to maintain in a larger system.
Navigate to the Example Project. You can be in any subfolder.
Select > Folder > Permissions.
Click the tab Project Groups.
There are 5 predefined project groups. See at a glance how many members are in each.
Click the name to see the membership. Click Example Team Members and see the list of all our example users.
Click Done in the popup.
Click the Lab Team to see that the two members are the team lead and the lab technician.
Click the Study Team to see that the two members are the team lead and the study researcher.
Next we'll review how these groups are assigned different permissions within the project's subfolders.
Permission Roles
Permission roles grant different types of access to a resource. Read, Edit, and Admin are typical examples; there are many more permission roles available in LabKey Server. Learn more here.
If you navigated away after the previous section, select > Folder > Permissions.
Click the Permissions tab.
In the left column list of folders, you will see the entire project hierarchy. The folder you are viewing is shown in bold. Click Example Project to see the permissions in the project itself.
The "Admin Team" is both project and folder administrator, and the "Example Team Members" group are Editors in the project container.
Click Collaboration Workspace and notice that the "Example Team Members" are editors here, too.
Click Laboratory Data. In this folder, the "Lab Team" group has editor permissions, and the "Example Team Members" group only has reader permission.
Note that when users are members of multiple groups, like in the case of our sample "lab_technician@local.test", they hold the sum of permissions granted through the groups they belong to. This lab_technician has read access with example team membership, but also editor access because of lab team membership, so that user will be able to edit contents here.
To see the user membership of any group, click the group name in the permissions UI.
To see all permissions granted to a given user, click the Permissions link in the group membership popup.
This example lab technician can edit content in the example project, the collaboration workspace folder, and the laboratory data folder. They can read but not edit content in the research study folder.
Close any popups, then click Cancel to exit the permissions editing UI.
Using impersonation, an admin can see what another user would be able to see and do on the server.
Navigate to the Example Project/Research Study folder using the project menu.
Notice that as yourself, the application admin on this server, you can see a (pencil) icon in the header of the Study Overview web part. You would click it to edit study properties. You also see the Manage tab.
Select (User) > Impersonate > User.
Select "lab_technician@local.test" from the dropdown and click Impersonate.
Now you are seeing the content as the lab technician would: with permission to read but not edit, as we saw when reviewing permissions above.
Notice the (pencil) icon and Manage tabs are no longer visible. You also no longer have some of the original options on the menu in the header.
Click Stop Impersonating to return to your own "identity".
Impersonate a Group
Impersonation of a group can help you better understand permissions and access. In particular, when configuring access and deciding what groups should include which users, group impersonation can be very helpful.
Navigate to the Example Project/Research Study folder, if you navigated away.
Select (User) > Impersonate > Group.
Choose the "Study Team" and click Impersonate.
Hover over the project menu and notice that the only folder in "Example Project" that members of the "Study Team" can read is this "Research Study" folder.
Click Stop Impersonating.
To see an error, select (User) > Impersonate > Group, choose the "Lab Team" and click Impersonate. This group does not have access to this folder, so the error "User does not have permission to perform this operation" is shown.
Click Stop Impersonating.
Impersonate a Role
You can also directly impersonate roles like "Reader" and "Submitter" to see what access those roles provide.To learn more about impersonation, see Test Security Settings by Impersonation.
Audit Logging
Actions on LabKey Server are extensively logged for audit and security review purposes. Impersonation is among the events logged. Before you complete this step, be sure you have stopped impersonating.
Select > Site > Admin Console.
Under Management, click Audit Log.
Using the pulldown, select User Events.
You will see the impersonation events you just performed. Notice that for impersonations of individual users, these are paired events - both the action taken by the user to impersonate, and the action performed "on" the user who was impersonated.
Explore other audit logs to see other kinds of events tracked by the server.
As a first project, let's create a "Tutorials" project in which you can run some LabKey tutorials.
Create a Project
To open the project creation wizard you can:
Click the "Create" button on the Trial Server home page.
OR: Select > Site > Create Project.
OR: Click the "Create Project" icon at the bottom of the project menu as shown:
The project creation wizard includes three steps.
Give the project a Name and choose the folder type.
Configure users and permissions (or accept the defaults and change them later).
Choose optional project settings (or configure them later).
Select > Site > Create Project.
Step 1: Create Project:
Enter the Name: "Tutorials". Project names must be unique on the server.
Leave the box checked to use this as the display title. If you unchecked it, you could enter a different display title.
Folder Type: Leave the default selection of "Collaboration". Other folder types available on your server are listed; hover to learn more about any type. Click Folder Help for the list of folder types available in a full LabKey installation.
Click Next.
Step 2: Users / Permissions: Choose the initial security configuration. As the admin, you can also change it later.
The default option "My User Only" lets you set up the project before inviting additional users.
The other option "Copy From Existing Project" is a helpful shortcut when creating new projects to match an existing one.
Leave "My User Only" selected and click Next.
Step 3: Project Settings:
On a LabKey Trial Server, you cannot Choose File Location, so this option is grayed out.
Advanced Settings are listed here for convenience, but you do not need to set anything here.
Simply click Finish to create your new project.
You'll now see the landing page of your new project and can start adding content.
Add Some Content
Let's customize the landing page and make it easier to use.
In the Wiki web part, click Create a new wiki page to display in this web part.
Leave the Name as "default"
Enter the Title "Welcome"
In the body field, enter: "Feel free to make a new subfolder and run a tutorial in it."
Click Save & Close.
Customize the page layout:
Select > Page Admin Mode.
In the web part header menu for your new "Welcome" wiki, select (triangle) > Move Up.
In the web part header menu for both "Messages" and "Pages", select (triangle) > Remove From Page.
Click Exit Admin Mode.
You can now use this project as the base for tutorials.
Subfolders Web Part
Most LabKey tutorials begin with creation of a new subfolder of a specific type, which you can add by clicking Create New Subfolder here. There are no subfolders to display yet, but once you add a few, this web part will look something like this, giving you a one click way to return to a tutorial folder later.
Now that you have created a new project and learned more about LabKey Server, consider sharing with a colleague. To invite someone to explore the Trial Server you created, simply add them as a new user:
Select > Site > Site Users.
Click Add Users above the grid of existing site users.
Enter one or more email addresses, each on it's own line.
If you want to grant the new user(s) the same permissions on the server as another user, such as yourself, check the box for Clone permissions from user: and enter the user ID. See what permissions will be cloned by clicking Permissions.
Notification email will be sent inviting the new user unless you uncheck this option. You can also see this email by clicking the here link in the UI that will appear.
Click Done.
You will see the new user(s) listed in the grid. Click the Permissions link for each one in turn and see the permissions that were granted. To change these in any given project or folder, navigate to it and use > Folder > Permissions. See Configure Permissions for more information.
What Else Can I Do?
Change the Color Scheme
The color scheme, or theme, is a good way to customize the look of a given project. To see what the other themes look like, see Web Site Theme.To change what your Tutorials project looks like:
Navigate to the Tutorials project, or any subfolder of it.
Select > Folder > Project Settings.
Under Theme, select another option.
Scroll down and click Save.
Return to the folder and see the new look.
Switch to the home project (click the LabKey logo in the upper left) and see the site default still applies there.
To restore the original look, return to the Tutorials project and reset the "Leaf" theme.
You can also change the theme for the site as a whole by selecting > Site > Site Console. Click Settings, then under Configuration, choose Look and Feel Settings. The same options are available for the Theme setting.
What's Next?
You can create more new projects and folders on your LabKey Trial Server to mock up how it might work for your specific solution. Many features available in a full LabKey installation are not shown in the trial version; you can learn more in our documentation.Feel free to contact us and we can help you determine if LabKey is the right fit for your research.
Extending Your Trial
When your LabKey Server Hosted Trial is nearing it's expiration date, you will see a banner message in the server offering you upgrade options. If you need a bit more time to explore, you can extend your trial beyond the initial 30 days.
From within your trial server, select > Manage Hosted Server Account.
Your LabKey Server trial provides a cloud-based environment where you can explore key features of the LabKey platform. Explore a preloaded example project, upload your own data, and try features only available in premium editions. Your trial lasts 30 days and we're ready to help you with next steps toward incorporating LabKey Server into your research projects.
LabKey provides a customizable framework for longitudinal study data. A study tracks information about subjects over time. You can explore a read-only example here:
This topic assumes you have your own time series data about some study subjects and helps you create your own study where you can upload and analyze data of your own. To do that, you will use a trial version of LabKey Server. It takes a few minutes to set up and you can use it for 30 days.
You may already have data you plan to upload. If not, we will provide example files to get you started.
Identify your Study Subjects
The subjects being studied, "participants" by default, can be humans, animals, trees, cells, or anything else you want to track.Look at a dataset you plan to use. How are subjects under study identified. What column holds identifying information?
Decide How To Track Time
Time Based:
Some studies track time using dates. The date of each data collection is entered and the study is configured to break these into month-long timepoints, labeled as "Month 1", "Month 2" etc.Visit Based:
You can also track time in a visit-by-visit manner. If the elapsed time between events is not relevant but you have a fixed sequence of visit events, a visit-based study may be appropriate. Visits are identified by sequence numbers.Continuous:
A continuous study is used when there is no strong concept of visits or fixed buckets of time. Data is entered and tracked by date without using timepoints.
Security
Identify your security requirements and how you plan to use the data in your study. Who will need access to what data. Will you want to be able to assign access 'per dataset' or is folder level security sufficient?When you select your Security Mode, use basic if folder-level is sufficient and "custom" otherwise. Decide if you want datasets to be editable by any non-admins after import. Learn more about study security in this topic: Manage Study Security
Create Your Study
Once your trial server has been created, you will receive an email notification with a direct link to it.
Click the Create Project button on the home page.
Enter Name "Tutorials", leave the default folder type and click Next, then Next and then Finish to create the new project with all the default settings.
In the new project, click Create New Subfolder.
Enter the Name you want. In our example, it is named "My New Study".
Select the folder type Study and click Next.
Click Finish to create the folder.
Click Create Study.
Under Look and Feel Properties:
Study Label: Notice your folder name has the word "Study" added; you can edit to remove redundancy here if you like.
Subject Noun (Singular and Plural): If you want to use a word other than "Participant/Participants" in the user interface, enter other nouns here.
Subject Column Name: Look at the heading in your dataset for the column containing identifiers. Edit this field to match that column.
Under Visit/Timepoint Tracking:
Select the type of time tracking you will use. Dates, Assigned Visits, or Continuous.
If needed for your selection, enter Start Date and/or Default Timepoint Duration.
Under Security:
Select whether to use basic or custom security and whether to make datasets editable or read-only.
To maximize the configurability of this study, choose Custom security with editable datasets.
Click Create Study.
You will now be on the Manage tab of your own new study. This tab exists in the read-only exploration study we provided, but you cannot see it there. It is visible only to administrators.
Upload Your Data
Add Demographic Data
Begin with your demographic dataset. Every study is required to have one, and it contains one row per study participant (or subject). It does not have to be named "Demographics.xls", and you do not have to name your dataset "Demographics"; that is just the convention used in LabKey examples and tutorials.
On the Manage tab of your new study, click Manage Datasets.
Click Create New Dataset.
Give the dataset a short name (such as "Demographics"). This name does not have to match your filename.
Under Data Row Uniqueness, select Participants Only (demographic data).
Click the Fields section to open it.
Drag your demographic data spreadsheet into the Import or infer fields from file target area.
The column names and data types will be inferred. You can change types or columns as needed.
In the Column Mapping section, be sure that your mappings are as intended: Participant (or Subject) column and either "Date" for date-based or "Sequence Num" for visit-based studies.
Note: All your study data is assumed to have these two columns.
Notice the Import data from this file upon dataset creation? section is enabled and preview the first few lines of your data.
Click Save to create and populate the dataset.
Click View Data to see it in a grid.
You can now click the Participants/Subjects tab in your study and see the list gleaned from your spreadsheet. Each participant has a built in participant report that will contain (so far) their demographic data.
Add More Datasets
Repeat the above process for the other data from your study, except under Data Row Uniqueness, non-demographic datasets will use the default Participants and Visits. For example, you might have some sort of Lab Results for a set of subjects. Each row in a dataset like this will have one row per participant and visit/date combination. This enables studying results over time.
For each data spreadsheet:
Select Manage > Manage Datasets > Create New Dataset.
Name each dataset, select the data row uniqueness, upload the file, confirm the fields, and import.
Once you have uploaded all your data, click the Clinical and Assay Data tab to see the list of datasets.
Join your Data
With only two datasets, one demographic, you can begin to integrate your data.
Click the Clinical and Assay Data tab.
Click the name of a non-demographic dataset, such as one containing "Lab Results". You will see your data.
Select (Grid Views) > Customize Grid.
You will see a Datasets node under Available Fields.
Click the to expand it.
Click the for the Demographics dataset.
Check one or more boxes to add Demographics columns to your grid view.
Click View Grid to see the joined view. Save to save it either as the default or as a named grid view.
To get started using LabKey Biologics LIMS, you can request a trial instance of Biologics LIMS. Go here to tell us more about your needs and request your trial.Trial instances contain some example data to help you explore using LabKey Biologics for your own research data. Your trial lasts 30 days and we're ready to help you understand how LabKey Biologics can work for you.
To get started using LabKey products, you can contact us to tell us more about your research and goals. You can request a customized demo so we can help understand how best to meet your needs. In some cases we will encourage you to evaluate and explore with your own data using a custom trial instance. Options include:
LabKey Server Trial
LabKey Server Trial instances contain a core subset of features, and sample content to help get you started. Upload your own data, try tutorials, and even create a custom site tailored to your research and share it with colleagues. Your trial lasts 30 days and we're ready to help you with next steps toward incorporating LabKey Server into your research projects.
Try the core features of LabKey Sample Manager using our example data and adding your own. Your trial lasts 30 days and we're ready to help you with next steps.
Try the core features of LabKey Biologics LIMS using our example data and tutorial walkthroughs. Your trial lasts 30 days and we're ready to help you with next steps toward incorporating LabKey Biologics into your work.
Tutorials provide a "hands on" introduction to the core features of LabKey Server, giving step-by-step instructions for building solutions to common problems. They are listed roughly from simple to more complex, and you can pick and choose only those that interest you.
This topic covers the quickest and easiest way to set up to run LabKey tutorials using a trial of LabKey Server. If you want to run tutorials not supported in the trial environment, or already have access to LabKey Server, see Set Up for Tutorials: Non-Trial.
Tutorials you can run using a free trial of LabKey Server are marked with this badge.
LabKey Trial Server
To run the LabKey Tutorials, you need three things:1. An account on a running LabKey Server instance. After contacting us about your specific research needs and goals, we may set up a LabKey Trial Server for you:
This topic explains how to set up for the LabKey Tutorials on a non-trial instance of the server. If you have access to a LabKey trial server and the tutorial you want will work there, use this topic instead: Set Up for Tutorials: Trial.
1. & 2. If your organization is already using LabKey Server, contact your administrator about obtaining access as an administrator to a project or folder to use. For example, they might create and assign you a "Tutorials-username" project or subfolder. They will provide you the account information for signing in, and the URL of the location to use.
LabKey Server installations in active use may have specialized module sets or other customizations which cause the UI to look different than tutorial instructions. It's also possible that you will not have the same level of access that you would have if you installed a local demo installation.
3. Learn the navigation and UI basics in this topic:
The project and folder hierarchy is like a directory tree and forms the basic organizing structure inside LabKey Server. Everything you create or configure in LabKey Server is located in some folder. Projects are the top level folders, with all the same behavior, plus some additional configuration options; they typically represent a separate team or research effort.The Home project is a special project. It is the default landing page when users log in and cannot be deleted. You can customize the content here to suit your needs. To return to the home project at any time, click the LabKey logo in the upper left corner.The project menu is on the left end of the menu bar and includes the display name of the current project.Hover over the project menu to see the available projects, and folders within them. Click any project or folder name to navigate there.Any project or folder with subfolders will show / buttons for expanding and contracting the list shown. If you are in a subfolder, there will be a clickable 'breadcrumb' trail at the top of the menu for quickly moving up the hierarchy. The menu will scroll when there are enough items, with the current location visible and expanded by default.The project menu always displays the name of the current project, even when you are in a folder or subfolder. A link with the Folder Name is shown near the top of page views like the following, offering easy one click return to the main page of the folder.For more about projects, folders, and navigation, see Project and Folder Basics.
Tabs
Using tabs within a folder can give you new "pages" of user interface to help organize content. LabKey study folders use tabs as shown here:When your browser window is too narrow to display tabs arrayed across the screen, they will be collapsed into a pulldown menu showing the current tab name and a (chevron). Click the name of the tab on this menu to navigate to it.For more about adding and customizing tabs, see Use Tabs.
Web Parts
Web parts are user interface panels that can be shown on any folder page or tab. Each web part provides some type of interaction for users with underlying data or other content.There is a main "wide" column on the left and narrower column on the right. Each column supports a different set of web parts. By combining and reordering these web parts, an administrator can tailor the layout to the needs of the users.To learn more, see Add Web Parts and Manage Web Parts. For a list of the types of web parts available in a full installation of LabKey Server, see the Web Part Inventory.
Header Menus and URLs
In the upper right, icon menus offer:
(Search): Click to open a site-wide search box
(Admin/Settings): Shown only to Admins: Administrative options available to users granted such access. See Admin Console for details about options available.
(User): Login and security options; help links to documentation and support options.
Watch the URL at the top of the page as you navigate LabKey Server and explore features on your server. Many elements are encoded in the URL and programmatic access via building URLs is possible with APIs. Learn more here: LabKey URLs.
Security Model
LabKey Server has a group and role-based security model. Whether an individual is authorized to see a resource or perform an action is checked dynamically based on the groups they belong to and roles (permissions) granted to them. Learn more here: Security.
Community Edition: Free to download and use forever. Best suited for technical enthusiasts and evaluators in non-mission-critical environments. LabKey provides documentation and you can request help using the support menu link on this page.
Premium Editions: Paid subscriptions that provide additional functionality and resources to help teams optimize workflows, manage complex projects, and explore multi-dimensional data. Premium Editions also include professional support services for the long-term success of your informatics solutions.
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
User and Administrator Training
LabKey's user and administrator training course, LabKey Fundamentals, is included with Premium Editions of LabKey Server. It provides an introduction to the following topics:
LabKey Server Basics: Explains the basic anatomy/architecture of the server and its moving parts. It outlines the basic structures of folders and data containers, and the modules that process requests and craft responses. Best practices for configuring folders is included. The role of Administrators is also described.
Security: Describes LabKey Server's role-based security model--and how to use it to protect your data resources. General folder-level security is described, as well as special security topics, such as dataset-level security and Protected Health Information (PHI) features. Practical security information is provided, such as how to set up user accounts, assigning groups and roles, best practices, and testing security configurations using impersonation.
Collaboration: Explains how to use the Wiki, Issues, and Messages modules. Branding and controlling the look-and-feel of your server are also covered.
Data Storage: Files and Database: Explains the two basic ways that LabKey Server can hold data: (1) as files and (2) as records in a database. Topics include: full-text search, converting tabular data files into database tables, special features of the LabKey database (such as 'lookups'), the role of SQL queries, adding other databases as external data sources.
Instrument Data: Explains how LabKey Server models and captures instrument-derived data, including how to create a new assay "design" from scratch, or how to use a prepared assay design. Special assay topics are covered, such as transform scripts, creating new assay design templates ("types") from simple configuration files, and how to replace the default assay user interface.
Clinical/Research Study Data Management: Explains how to integrate heterogeneous data, such as instrument, clinical, and demographic data, especially in the context of longitudinal/cohort studies.
Reports: Explains the various ways to craft reports on your data, including R reports, JavaScript reports, and built-in visualizations, such as Time Charts, Box Plots, and Scatter Plots.
Samples: Explains how to manage sample data and link it to assay and clinical data.
Advanced Topics: A high-level overview of how to extend LabKey Server. The Starter Edition includes support for users writing custom SQL, R scripts, and ETLs. The Professional Edition provides support for users extending LabKey Server with JavaScript/HTML client applications, user-created file modules.
Developer Training
LabKey's developer training is included in the Professional and Enterprise Editions of LabKey Server. It is tailored to your project's specific needs and can cover:
Premium Resources AvailableSubscribers to premium editions of LabKey Server can get a head start with video and slide deck training resources available in this topic:
LabKey Server is at it's core an open-source software platform designed to help research organizations integrate, analyze, and share complex biomedical data. Adaptable to varying research protocols, analysis tools, and data sharing requirements, LabKey Server couples the flexibility of a custom solution with enterprise-level scalability to support scientific workflows.
This topic is for absolute beginners to LabKey Server. It explains what LabKey Server is for, how it works, and how to build solutions using its many features.
LabKey Server's features can be grouped into three main areas:
1. Data Repository
LabKey Server lets you bring data together from multiple sources into one repository. These sources can be physically separated in different systems, such as data in Excel spreadsheets, different databases, REDCap, etc. Or the data sources can be separated "morphologically", having different shapes. For example, patient questionnaires, instrument-derived assay data, medical histories, and sample inventories all have different data shapes, with different columns names and different data types. LabKey Server can bring all of this data together to form one integrated whole that you can browse and analyze together.
2. Data Showcase
LabKey Server lets you securely present and highlight data over the web. You can present different profiles of your data to different audiences. One profile can be shown to the general public with no restrictions, while another profile can be privately shared with selected individual colleagues. LabKey Server lets you collaborate with geographically separated teams, or with your own internal team members. In short, LabKey Server lets you create different relationships between data and audiences, where some data is for general viewing, other data is for peer review, and yet other data is for group editing and development.
3. Electronic Laboratory
LabKey Server provides many options for analyzing and inquiring into data. Like a physical lab that inquires into materials and natural systems, LabKey Server makes data itself the object of inquiry. This side of LabKey Server helps you craft reports and visualizations, confirm hypotheses, and generally provide new insights into your data, insights that wouldn't be possible when the data is separated in different systems and invisible to other collaborators.
The LabKey Server Platform
LabKey Server is a software platform, as opposed to an application. Applications have fixed use cases targeted on a relatively narrow set of problems. As a platform, LabKey Server is different: it has no fixed use cases, instead it provides a broad range of tools that you configure to build your own solutions. In this respect, LabKey Server is more like a car parts warehouse and not like any particular car. Building solutions with LabKey Server is like building new cars using the car parts provided. To build new solutions you assemble and connect different panels and analytic tools to create data dashboards and workflows.The following illustration shows how LabKey Server takes in different varieties of data, transforms into reports and insights, and presents them to different audiences.
How Does LabKey Server Work?
LabKey Server is a web server, and all web servers are request-response machines: they take in requests over the web (typically as URLs through a web browser) and then craft responses which are displayed to the user.
Modules
Modules are the main functionaries in the server. Modules interpret requests, craft responses, contain all of the web parts, and application logic. The responses can take many different forms:
a web page in a browser
an interactive grid of data
a report or visualization of underlying data
a file download
a long-running calculation or algorithm
LabKey Server uses a PostgreSQL database as its primary data store. You can attach other external databases to the server. Learn about external data sources here:
LabKey Server offers non-disruptive integration with your existing systems and workflows. You can keep your existing data systems in place, using LabKey Server to augment them, or you can use LabKey Server to replace your existing systems. For example, REDCap to collect patient data, and an external repository to hold medical histories, LabKey Server can synchronize and combine the data in these systems, so you can build a more complete picture of your research results, without disrupting the workflows you have already built.The illustration below shows the relationships between web browsers, LabKey Server, and the underlying databases. The modules shown are not a complete set; many other modules are included in LabKey Server.
User Interface
You configure your own user interface by adding panels, aka "web parts", each with a specific purpose in mind. Some example web parts:
The Wiki web part displays text and images to explain your research goals and provide context for your audience. (The topic you are reading right now is displayed in a Wiki web part.)
The Files web part provides an area to upload, download, and share files will colleagues.
The Query web part displays interactive grids of data.
The Report web part displays the results of an R or JS based visualization.
Group web parts on separate tabs to form data dashboards.The illustration below shows a data dashboard formed from tabs and web parts.
Folders and Projects
Folders are the "blank canvases" of LabKey Server, the workspaces where you organize dashboards and web parts. Folders are also important in terms of securing your data, since you grant access to audience members on a folder-by-folder basis. Projects are top level folders: they function like folders, but have a wider scope. Projects also form the center of configuration inside the server, since any setting made inside a project cascades into the sub-folders by default.
Security
LabKey uses "role-based" security to control who has access to data. You assign roles, or "powers", to each user who visits your server. Their role determines how much they can see and do with the data. The available roles include: Administrator (they can see and do everything), Editors, Readers, Submitters, and others. Security is very flexible in LabKey Server. Any security configuration you can imagine can be realized: whether you want only a few select individual to see your data, or if you want the whole world to see your data.The server also has extensive audit logs built. The audit logs record:
Who has logged in and when
Changes to a data record
Queries performed against the database
Server configuration changes
File upload and dowload events
And many other activities
The Basic Workflow: From Data Import to Reports
To build solutions with LabKey Server, follow this basic workflow: import or synchronize your data, apply analysis tools and build reports on top of the data, and finally share your results with different audiences. Along the way you will add different web parts and modules as needed. To learn the basic steps, start with the tutorials, which provide step-by-step instructions for using the basic building blocks available in the server.
Ready to See More?
You don't need to download or install anything to try a few basic features right now: Try it Now: Data Grids.
- Admin/Settings: Administrative options available to users granted such access.
- User: Login and security options; context-sensitive help.
Product Selection Menu (Premium Feature)
LabKey offers several products, all of which may be integrated on a single Premium Edition of LabKey Server. When more than one application is installed on your server, you can use the menu to switch between them in a given container.
Learn more in this topic: Product Selection Menu
Project and Folder Menu
Hover over the project menu to see the available projects and folders within them. Note that only locations in which you have at least "Read" access are shown. If you have access to a subfolder, but not to the parent, the name of the parent folder or project will still be included on the menu for reference.Any project or folder with subfolders will show and buttons for expanding and contracting the list shown. The menu will scroll when there are enough items, and the current location will be visible and expanded by default.
- Permalink URL: Click for a permalink to the current location.
Administrators have additional options shown in the bottom row of this menu.
Navigation
Click the name of any project or folder in the menu to navigate to that location. When you are in a folder or subfolder, you will see a "breadcrumb" path at the top of the projects menu, making it easy to step back up one or more levels. Note that if you have access to a subfolder, but not to one or more of its parent folders, the parent folder(s) will be still be displayed in the menu (and on the breadcrumb path) but those locations will not be clickable links.Notice that from within a folder or subfolder, the project menu still displays the name of the project. A link with the Folder Name is shown next to the title offering easy one click return to the main page of the folder.
Context Sensitive Help
Click the icon in the upper right to open the user menu. Options for help include:
Support: Options for asking questions or learning more about whichever LabKey product you are using.
LabKey Documentation: Navigates directly to the relevant documentation for the product you are using. In some cases, you'll see specific documentation to the feature area you are viewing when you click.
LabKey Server lets you explore, interrogate and present your biomedical research data online and interactively in a wide variety of ways. The topics in this section give you a broad overview of data basics.
LabKey Server offers a wide variety of ways to store and organize data. Different data structure types offer specific features, which make them more or less suited for specific scenarios. You can think of these data structures as "table types", each designed to capture a different kind of research data.This topic reviews the data structures available within LabKey Server, and offers guidance for choosing the appropriate structure for storing your data.
The primary deciding factors when selecting a data structure will be the nature of the data being stored and how it will be used. Information about lab samples should likely be stored as a sample type. Information about participants/subjects/animals over time should be stored as datasets in a study folder. Less structured data may import into LabKey Server faster than highly constrained data, but integration may be more difficult. If you do not require extensive data integration or specialized tools, a more lightweight data structure, such as a list, may suit your needs.The types of LabKey Server data structures appropriate for your work depend on the research scenarios you wish to support. As a few examples:
Management of Simple Tabular Data. Lists are a quick, flexible way to manage ordinary tables of data, such as lists of reagents.
Integration of Data by Time and Participant for Analysis. Study datasets support the collection, storage, integration, and analysis of information about participants or subjects over time.
Analysis of Complex Instrument Data. Assays help you to describe complex data received from instruments, generate standardized forms for data collection, and query, analyze and visualize collected data.
These structures are often used in combination. For example, a study may contain a joined view of a dataset and an assay with a lookup into a list for names of reagents used.
Universal Table Features
Most LabKey data structures support the following features:
Interactive, Online Grids
Data Validation
Visualizations
SQL Queries
Lookup Fields
Lists
Lists are the simplest and least constrained data type. They are generic, in the sense that the server does not make any assumptions about the kind of data they contain. Lists are not entirely freeform; they are still tabular data and have primary keys, but they do not require participant IDs or time/visit information. There are many ways to visualize and integrate list data, but some specific applications will require additional constraints.Lists data can be imported in bulk as part of a TSV, or as part of a folder, study, or list archive. Lists also allow row-level insert/update/delete.Lists are scoped to a single folder, and its child workbooks (if any).
Assays
Assays capture data from individual experiment runs, which usually correspond to an output file from some sort of instrument. Assays have an inherent batch-run-results hierarchy. They are more structured than lists, and support a variety of specialized structures to fit specific applications. Participant IDs and time information are required.Specific assay types are available, which correspond to particular instruments and offer defaults specific to use of the given assay instrument. Results schema can range from a single, fixed table to many interrelated tables. All assay types allow administrators to configure fields at the run and batch level. Some assay types allow further customization at other levels. For instance, the Luminex assay type allows admins to customize fields at the analyte level and the results level. There is also a general purpose assay type, which allows administrators to completely customize the set of result fields.Usually assay data is imported from a single data file at a time, into a corresponding run. Some assay types allow for API import as well, or have customized multi-file import pathways. Assays result data may also be integrated into a study by aligning participant and time information, or by sample id.Assay designs are scoped to the container in which they are defined. To share assay designs among folders or subfolders, define them in the parent folder or project, or to make them available site-wide, define them in the Shared project. Run and result data can be stored in any folder in which the design is in scope.
Datasets
Clinical Datasets are designed to capture the variable characteristics of an organism over time, like blood pressure, mood, weight, and cholesterol levels. Anything you measure at multiple points in time will fit well in a Clinical Dataset.Datasets are always part of a study. They have two required fields:
ParticipantId (this name may vary) - Holds the unique identifier for the study subject.
Date or Visit (the name may vary) - Either a calendar date or a number.
There are different types of datasets with different cardinality, also known as data row uniqueness:
Demographic: Zero or one row for each subject. For example, each participant has only one enrollment date.
“Standard”/"Clinical": Can have multiple rows per subject, but zero or one row for each subject/timepoint combination. For example, each participant has exactly one weight measurement at each visit.
“Extra key”/"Assay": can have multiple rows for each subject/timepoint combination, but have an additional field providing uniqueness of the subject/timepoint/arbitrary field combination. For example, many tests might be run each a blood sample collected for each participant at each visit.
Datasets have special abilities to automatically join/lookup to other study datasets based on the key fields, and to easily create intelligent visualizations based on these sorts of relationships.A dataset can be backed by assay data that has been copied to the study. Behind the scenes, this consists of a dataset with rows that contain the primary key (typically the participant ID) of the assay result data, which is looked up dynamically.Non-assay datasets can be imported in bulk (as part of a TSV paste or a study import), and can also be configurable to allow row-level inserts/updates/deletes.Datasets are typically scoped to a single study in a single folder. In some contexts, however, shared datasets can be defined at the project level and have rows associated with any of its subfolders.Datasets have their own study security configuration, where groups are granted access to datasets separately from their permission to the folder itself. Permission to the folder is a necessary prerequisite for dataset access (i.e., have the Reader role for the folder), but is not necessarily sufficient.A special type of dataset, the query snapshot, can be used to extract data from some other sources available in the server, and create a dataset from it. In some cases, the snapshot is automatically refreshed after edits have been made to the source of the data. Snapshots are persisted in a physical table in the database (they are not dynamically generated on demand), and as such they can help alleviate performance issues in some cases.
Custom Queries
A custom query is effectively a non-materialized view in a standard database. It consists of LabKey SQL, which is exposed as a separate, read-only query/table. Every time the data in a custom query is used, it will be re-queried from the database.In order to run the query, the current user must have access to the underlying tables it is querying against.Custom queries can be created through the web interface in the schema browser, or supplied as part of a module.
Sample Types
Sample types allow administrators to create multiple sets of samples in the same folder, which each have a different set of customizable fields.Sample types are created by pasting in a TSV of data and identifying one, two, or three fields that comprise the primary key. Subsequent updates can be made via TSV pasting (with options for how to handle samples that already exist in the set), or via row-level inserts/updates/deletes.Sample types support the notion of one or more parent sample fields. When present, this data will be used to create an experiment run that links the parent and child samples to establish a derivation/lineage history. Samples can also have "parents" of other dataclasses, such as a "Laboratory" data class indicating where the sample was collected.One sample type per folder can be marked as the “active” set. Its set of columns will be shown in Customize Grid when doing a lookup to a sample table. Downstream assay results can be linked to the originating sample type via a "Name" field -- for details see Samples.Sample types are resolved based on the name. The order of searching for the matching sample type is: the current folder, the current project, and then the Shared project. See Shared Project.
DataClasses
DataClasses can be used to capture complex lineage and derivation information, for example, the derivations used in bio-engineering systems. Examples include:
Reagents
Gene Sequences
Proteins
Protein Expression Systems
Vectors (used to deliver Gene Sequences into a cell)
Constructs (= Vectors + Gene Sequences)
Cell Lines
You can also use dataclasses to track the physical and biological Sources of samples.
Similarities with Sample Types
A DataClass is similar to a Sample Type or a List, in that it has a custom domain. DataClasses are built on top of the exp.Data table, much like Sample Types are built on the exp.Materials table. Using the analogy syntax:
SampleType : exp.Material :: DataClass : exp.Data
Rows from the various DataClass tables are automatically added to the exp.Data table, but only the Name and Description columns are represented in exp.Data. The various custom columns in the DataClass tables are not added to exp.Data. A similar behavior occurs with the various Sample Type tables and the exp.Materials table.Also like Sample Types, every row in a DataClass table has a unique name, scoped across the current folder. Unique names can be provided (via a Name or other ID column) or generated using a naming pattern.For more information, see Data Classes.
Domains
A domain is a collection of fields. Lists, Datasets, SampleTypes, DataClasses, and the Assay Batch, Run, and Result tables are backed by an LabKey internal datatype known as a Domain. A Domain has:
an ordered set of fields along with their properties.
Each Domain type provides specialized handling for the domains it defines. The number of domains defined by a data type varies; for example, Assays define multiple domains (batch, run, etc.), while Lists and Datasets define only one domain each.The fields and properties of a Domain can be edited interactively using the field editor or programmatically using the JavaScript LABKEY.Domain APIs.Also see Modules: Domain Templates.
Domain/Data Structure Names
Data structures (Domains) like Sample Types, Source Types, Assay Designs, etc. must have unique names and avoid specific special characters, particularly if they are to be used in naming patterns or API calls. Names must follow these rules:
Must not be blank
Must start with a letter or a number character.
Must contain only valid unicode characters. (no control characters)
May not contain any of these characters:
<>[]{};,`"~!@#$%^*=|?\
May not contain 'tab', 'new line', or 'return' characters.
May not contain space followed by dash followed by a character.
i.e. these are allowed: "a - b" or "a-b" or "a–-b"
these are not allowed: "a -b", "a –-b"
For domains that support naming expressions (Sample Types, Sources), these special substitution strings are not allowed to be used as names:
While field names themselves are allowed to contain special characters, it is best practice to have "database legal" names without them and use the special characters you want to show users in the field label.You can find a list of all the field names using the "exp.Fields" table in the query browser. View data, use the grid folder filter to set the desired scope, then filter the Special Characters column for "true".
External Schemas
External schemas allow an administrator to expose the data in a "physical" database schema through the web interface, and programmatically via APIs. They assume that some external process has created the schemas and tables, and that the server has been configured to connect to the database.Administrators have the option of exposing the data as read-only, or as insert/update/delete. The server will auto-populate standard fields like Modified, ModifiedBy, Created, CreatedBy, and Container for all rows that it inserts or updates. The standard bulk option (TSV, etc) import options are supported.External schemas are scoped to a single folder. If an exposed table has a "Container" column, it will be filtered to only show rows whose values match the EntityId of the folder.The server can connect to a variety of external databases, including Oracle, MySQL, SAS, Postgres, and SQLServer. The schemas can also be housed in the standard LabKey Server database.The server does not support cross-database joins. It can do lookups (based on single-column foreign keys learned via JDBC metadata, or on XML metadata configuration) only within a single database though, regardless of whether it’s the standard LabKey Server database or not.Learn more in this topic:
Linked schemas allow you to expose data in a target folder that is backed by some other data in a different source folder. These linked schemas are always read-only.This provides a mechanism for showing different subsets of the source data in a different folder, where the user might not have permission (or need) to see everything else available in the source folder.The linked schema configuration, set up by an administrator, can include filters such that only a portion of the data in the source schema/table is exposed in the target.Learn more in this topic:
This topic explains how to best prepare your data for import so you can meet any requirements set up by the target data structure.LabKey Server provides a variety of different data structures for different uses: Assay Designs for capturing instrument data, Datasets for integrating heterogeneous clinical data, Lists for general tabular data, etc. Some of these data structures place strong constraints on the nature of the data to be imported, for example Datasets make uniqueness constraints on the data; other data structures, such as Lists, make few assumptions about incoming data.
Integer: A 4-byte signed integer that can hold values ranging -2,147,483,648 to +2,147,483,647.
Decimal (Floating Point): An 8-byte double precision floating point number that can hold very large and very small values. Values can range approximately 1E-307 to 1E+308 with a precision of at least 15 digits. As with most standard floating point representations, some values cannot be converted exactly and are stored as approximations. It is often helpful to set on Decimal fields a display format that specifies a fixed or maximum number of decimal places to avoid displaying approximate values.
General Advice: Avoid Mixed Data Types in a Column
LabKey tables (Lists, Datasets, etc.) are implemented as database tables. So your data should be prepared for insertion into a database. Most importantly, each column should conform to a database data type, such as Text, Integer, Decimal, etc. Mixed data in a column will be rejected when you try to upload it. WrongThe following table mixes Boolean and String data in a single column.
ParticipantId
Preexisting Condition
P-100
True, Edema
P-200
False
P-300
True, Anemia
Right Split out the mixed data into separate columns
ParticipantId
Preexisting Condition
Condition Name
P-100
True
Edema
P-200
False
P-300
True
Anemia
General Advice: Avoid Special Characters in Column Headers
Column names should avoid special characters such as !, @, #, $, etc.
Column names should contain only letters, numbers, spaces, and underscores; and should begin only with a letter or underscore. We also recommend underscores instead of spaces. WrongThe following table has special characters in the column names.
Participant #
Preexisting Condition?
P-100
True
P-200
False
P-300
True
RightThe following table removes the special characters and replaces spaces with underscores.
Participant_Number
Preexisting_Condition
P-100
True
P-200
False
P-300
True
Data Column Aliasing
Use data column aliasing to work with non-conforming data, meaning the provided data has different columns names or different value ids for the same underlying thing. Examples include:
A lab provides assay data which uses different participant ids than those used in your study. Using different participant ids is often desirable and intentional, as it provides a layer of PHI protection for the lab and the study.
Excel files have different column names for the same data, for example some files have the column "Immune Rating" and other have the column "Immune Score". You can define an arbitrary number of these import aliases to map to the same column in LabKey.
When importing data, if there are unrecognized fields in your spreadsheet, meaning fields that are not included in the data structure definition, they will be ignored. In some situations, such as when importing sample data, you will see a warning banner explaining that this is happening:
Data Format Considerations
Handling Backslash Characters
If you have a TSV or CSV file that contains backslashes in any text field, the upload will likely ignore the backslash and either substitute it for another value or ignore it completely. This occurs because the server treats backslashes as escape characters, for example, \n will insert a new line, \t will insert a tab, etc.To import text fields that contain backslashes, wrap the values in quotes, either by manual edit or by changing the save settings in your editor. For example:
Test\123
should be:
"Test\123"
Note that this does not apply to importing Excel files. Excel imports are handled differently, and text data within the cell is processed such that text is in quotes.
Clinical Dataset Details
Datasets are intended to capture measurements events on some subject, like a blood pressure measurement or a viral count at some point it time. So datasets are required to have two columns:
a subject id
a time point (either in the form of a date or a number)
Also, a subject cannot have two different blood pressure readings at a given point in time, so datasets reflect this fact by having uniqueness constraints: each record in a dataset must have a unique combination of subject id plus a time point. WrongThe following dataset has duplicate subject id / timepoint combinations.
ParticipantId
Date
SystolicBloodPressure
P-100
1/1/2000
120
P-100
1/1/2000
105
P-100
2/2/2000
110
P-200
1/1/2000
90
P-200
2/2/2000
95
RightThe following table removes the duplicate row.
ParticipantId
Date
SystolicBloodPressure
P-100
1/1/2000
120
P-100
2/2/2000
110
P-200
1/1/2000
90
P-200
2/2/2000
95
Demographic Dataset Details
Demographic datasets have all of the constraints of clinical datasets, plus one more: a given subject identifier cannot appear twice in a demographic dataset. WrongThe following demographic dataset has a duplicate subject id.
ParticipantId
Date
Gender
P-100
1/1/2000
M
P-100
1/1/2000
M
P-200
1/1/2000
F
P-300
1/1/2000
F
P-400
1/1/2000
M
RightThe following table removes the duplicate row.
ParticipantId
Date
Gender
P-100
1/1/2000
M
P-200
1/1/2000
F
P-300
1/1/2000
F
P-400
1/1/2000
M
Date Parsing Considerations
Whether to parse user-entered or imported date values as Month-Day-Year (as typical in the U.S.) or Day-Month-Year (as typical outside the U.S.) is set at the site level. For example, 11/7/2020 is either July 11 or November 7; a value like 11/15/2020 is only valid Month-Day-Year (US parsing).If you attempt to import date values with the "wrong" format, they will be interpreted as strings, so you will see errors similar to ""Could not convert value '11/15/20' (String) for Timestamp field 'FieldName'."
Use Import Templates
For the most reliable method of importing data, first obtain a template for the data you are importing. Most data structures will include a Download Template button when you select any bulk import method, such as importing from a file.Use the downloaded template as a basis for your import file. It will include all possible columns and will exclude unnecessary ones. You may not need to populate every column of the template when you import data.
Premium Feature AvailableSubscribers to Sample Manager, LabKey LIMS, and Biologics LIMS have access to download templates in even more places. Learn more in this Sample Manager documentation topic:
When bulk importing data (via > Import bulk data) the default Import Option is Add rows, for adding new rows only. If you include data for existing rows, the import will fail.To update data for existing rows, select the option to Update rows. Note that update is not supported for Lists with auto-incrementing integer keys or Datasets with system-managed third keys.
If you want to merge existing data updates with new rows, select Update rows, then check the box to Allow new rows during update. Note that merge is not supported for Lists with auto-incrementing integer keys or Datasets with system-managed third keys.
Learn more about updating and merging data in these topics:
In some contexts, such as creating a list definition and populating it at the same time and importing samples from file into Sample Manager or LabKey Biologics, you will see a few lines "previewing" the data you are importing.These data previews shown in the user interface do not apply field formatting from either the source spreadsheet or the destination data structure.In particular, when you are importing Date and DateTime fields, they are always previewed in ISO format (yyyy-MM-dd hh:mm) regardless of source or destination formatting. The Excel format setting is used to infer that this is a date column, but is not carried into the previewer or the imported data.For example, if you are looking at a spreadsheet in Excel, you may see the value with specific date formatting applied, but this is not how Excel actually stores the date value in the field. In the image below, the same 'DrawDate' value is shown with different Excel formatting applied. Date values are a numeric offset from a built-in start date. To see the underlying value stored, you can view the cell in Excel with 'General' formatting applied (as shown in the fourth line of the image below), though note that 'General' formatted cells will not be interpreted as date fields by LabKey. LabKey uses any 'Date' formatting to determine that the field is of type 'DateTime', but then all date values are shown in ISO format in the data previewer, i.e. here "2022-03-10 00:00".After import, display format settings on LabKey Server will be used to display the value as you intend.
OOXML Documents Not Supported
Excel files saved in the "Strict Open XML Spreadsheet" format will generate an .xlsx file, however this format is not supported by POI, the library LabKey uses for parsing .xlsx files. When you attempt to import this format into LabKey, you will see the error:
There was a problem loading the data file. Unable to open file as an Excel document. "Strict Open XML Spreadsheet" versions of .xlsx files are not supported.
As an alternative, open such files in Excel and save as the ordinary "Excel Workbook" format.
Learn more about this lack of support in this Apache issue.
This topic covers using the Field Editor, the user interface for defining and editing the fields that represent columns in a grid of data. The set of fields, also known as the schema or "domain", describes the shape of data. Each field, or column, has a main field type and can also have various properties and settings to control display and behavior of the data in that column.The field editor is a common tool used by many different data structures in LabKey. The method for opening the field editor varies by data structure, as can the types of field available. Specific details about fields of each data type are covered in the topic: Field Types and Properties.
How you open the field editor depends on the type of data structure you are editing.
Data Structure
Open for Editing Fields
List
Open the list and click Design in the grid header
Dataset
In the grid header, click Manage, then Edit Definition, then the Fields section
Assay Design
Select Manage Assay Design > Edit Assay Design, then click the relevant Fields section
Sample Type
Click the name of the type and then Edit Type
Data Class
Click the name of the class and then Edit
Study Properties
Go to the Manage tab and click Edit Additional Properties
Specimen Properties
Under Specimen Repository Settings, click Edit Specimen/Vial/Specimen Event Fields
User Properties
Go to > Site > Site Users and click Change User Properties
Issue Definitions
Viewing the issue list, click Admin
Query Metadata
Go to > Go To Module > Query, select the desired schema and query, then click Edit Metadata
If you are learning to use the Field Editor and do not yet have a data structure to edit, you can get started by creating a simple list as shown in the walkthrough of this topic.
In a folder where you can practice new features, select > Manage Lists.
Click Create New List.
Give the list a name and click the Fields section to open it.
Default System Fields for Sample Types and Data Classes
Both Sample Types and Data Classes show Default System Fields at the top of the Fields panel. Other data types do not show this section.
Some system fields, such as Name cannot be disabled and are always required. Checkboxes are inactive when you cannot edit them.
Other fields, including the MaterialExpDate (Expiration Date) column for Sample Types, can be adjusted using checkboxes, similar to the Description field for Data Classes, shown below.
Enabled: Uncheck the box in this column to disable a field. This does not prevent the field from being created, and the name is still reserved, but it will not be shown to users.
Note that if you disable a field, the data in it is not deleted, you could later re-enable the field and recover any past data.
Required: Check this box to make a field required.
Click the to collapse this section and move on to any Custom Fields as for any data type.
Both options are available when the set of fields is empty. Once you have defined some fields by either method, the manual editor is the only option for adding more fields.
Import or Infer Fields from File
When the set of fields is empty, you can import (or infer) new fields from a file.
The range of file formats supported depends on the data structure. Some can infer fields from a data file; all support import from a JSON file that contains only the field definitions.
When a JSON file is imported, all valid properties for the given fields will be applied. See below.
Click to select or drag and drop a file of an accepted format into the panel.
The fields inferred from the file will be shown in the manual field editor, where you may fine tune them or save them as is.
Note that if your file includes columns for reserved fields, they will not be shown as inferred. Reserved field names vary by data structure and will always be created for you.
Manually Define Fields
After clicking Manually Define Fields, the panel will change to show the manual field editor. In some data structures, you'll see a banner about selecting a key or associating with samples, but for this walkthrough of field editing, these options are ignored.
Depending on the type of data structure, and whether you started by importing some fields, you may see a first "blank" field ready to be defined.
If not, click Add Field to add one.
Give the field a Name. If you enter a field name with a space in it, or other special character, you will see a warning. It is best practice to use only letters, numbers and underscores in the actual name of a field. You can use the Label property to define how to show this field name to users.
SQL queries, R scripts, and other code are easiest to write when field names only contain combination of letters, numbers, and underscores, and start with a letter or underscore.
If you include a dot . in your field name, you may see unexpected behavior since that syntax is also used as a separator for describing a lookup field. Specifically, participant views will not show values for fields where the name includes a .
Use the drop down menu to select the Data Type.
The data types available vary based on the data structure. Learn which structures support which types here.
Each data type can have a different collection of properties you can set.
Once you have saved fields, you can only make limited changes to the type.
You can use the checkbox if you want to make it required that that field have a value in every row.
Continue to add any new fields you need - one for each column of your data.
To edit fields, reopen the editor and make the changes you need. If you attempt to navigate away with unsaved changes you will have the opportunity to save or discard them. When you are finished making changes, click Save.Once you have saved a field or set of fields, you can change the name and most options and other settings. However, you can only make limited changes to the type of a field. Learn about specific type changes in this section.
Rearrange Fields
To change field order, drag and drop the rows using the six-block handle on the left.
Delete Fields
To remove one field, click the on the right. It is available in both the collapsed and expanded view of each field and will turn red when you hover.To delete one or more, use the selection checkboxes to select the fields you want to delete. You can use the box at the top of the column to select all fields, and once any are selected you will also see a Clear button. Click Delete to delete the selected fields.You will be asked to confirm the deletion and reminded that all data in a deleted field will be deleted as well. Deleting a field cannot be undone.Click Save when finished.
Edit Field Properties and Options
Each field has a data type and can have additional properties defined. The properties available vary based on the field type. Learn more in this topic: Field Types and PropertiesTo open the properties for a field, click the icon (it will become a handle for closing the panel). For example, the panel for a text field looks like:Fields of all types have:
Most field types have a section for Type-specific Field Options: The details of these options as well as the specific kinds of conditional formatting and validation available for each field type are covered in the topic: Field Types and Properties.
Name and Linking Options
All types of fields allow you to set the following properties:
Description: An optional text description. This will appear in the hover text for the field you define. XML schema name: description.
Label: Different text to display in column headers for the field. This label may contain spaces. The default label is the Field Name with camelCasing indicating separate words. For example, the field "firstName" would by default be labelled "First Name". If you wanted to show the user "Given Name" for this field instead, you would add that string in the Label field.
Import Aliases: Define alternate field names to be used when importing from a file to this field. This option offers additional flexibility of recognizing an arbitrary number of source data names to map to the same column of data in LabKey. Multiple aliases may be separated by spaces or commas. To define an alias that contains spaces, use double-quotes (") around it.
URL: Use this property to change the display of the field value within a data grid into a link. Multiple formats are supported, which allow ways to easily substitute and link to other locations in LabKey. The ${ } syntax may be used to substitute another field's value into the URL. Learn more about using URL Formatting Options.
Ontology Concept:(Premium Feature) In premium editions of LabKey Server, you can specify an ontology concept this field represents. Learn more in this topic: Concept Annotations
Conditional Formatting and Validation Options
Most field types offer conditional formatting. String-based fields offer regular expression validation. Number-based fields offer range expression validation. Learn about options supported for each type of field here.
Note that conditional formatting is not supported in Sample Manager, unless used with a Premium Edition of LabKey Server. The LabKey LIMS and Biologics LIMS products both support conditional formats.
Create Conditional Format Criteria
Conditional formats change how data is displayed depending on the value of the data. Learn more in this topic:
To add a conditional format for a field where it is supported:
Click Add Format to open the conditional format editor popup.
Specify one or two Filter Type and Filter Value pairs.
Select Display Options for how to show fields that meet the formatting criteria:
Bold
Italic
Strikethrough
Text/Fill Colors: Choose from the picker (or type into the #000000 area) to specify a color to use for either or both the text and fill.
You will see a cell of Preview Text for a quick check of how the colors will look.
Add an additional format to the same field by clicking Add Formatting. A second panel will be added to the popup.
When you are finished defining your conditional formats, click Apply.
Create Regular Expression Validator
Click Add Regex to open the popup.
Enter the Regular Expression that this field's value will be evaluated against.
All regular expressions must be compatible with Java regular expressions as implemented in the Pattern class.
You can test your expression using a regex interpreter, such as https://regex101.com/.
Description: Optional description.
Error Message: Enter the error message to be shown to the user when the value fails this validation.
Check the box for Fail validation when pattern matches field value in order to reverse the validation: With this box unchecked (the default) the pattern must match the expression. With this box checked, the pattern may not match.
Name: Enter a name to identify this validator.
You can use Add Regex Validator to add a second condition. The first panel will close and show the validator name you gave. You can reopen that panel using the (pencil) icon.
Click Apply when your regex validators for this field are complete.
Click Save.
Create Range Expression Validator
Click Add Range to open the popup.
Enter the First Condition that this field's value will be evaluated against. Select a comparison operator and enter a value.
Optionally enter a Second Condition.
Description: Optional description.
Error Message: Enter the error message to be shown to the user when the value fails this validation.
Name: Enter a name to identify this validator.
You can use Add Range Validator to add a second condition. The first panel will close and show the validator name you gave. You can reopen that panel using the (pencil) icon.
Click Apply when your range validators for this field are complete.
Click Save.
Advanced Settings for Fields
Open the editing panel for any field and click Advanced Settings to access even more options:
Display Options: Use the checkboxes to control how and in which contexts this field will be available.
Show field on default view of the grid
Show on update form when updating a single row of data
Show on insert form when updating a single row of data
Show on details page for a single row
Default Value Options: Automatically supply default values when a user is entering information. Not available for Sample Type fields or for Assay Result fields.
Default Type: How the default value for the field is determined. Options:
Last entered: (Default) If a default value is provided (see below), it will be entered and editable for the user's first use of the form. During subsequent uploads, the user will see their last entered value.
Editable default: An editable default value will be entered for the user. The default value will be the same for every user for every upload.
Fixed value: Provides a fixed default value that cannot be edited.
Default Value: Click Set Default Values to set default values for all the fields in this section.
Default values are scoped to the individual folder. In cases where field definitions are shared, you can manually edit the URL to set default values in the specific project or folder.
Note that setting default values will navigate you to a new page. Save any other edits to fields before leaving the field editor.
Miscellaneous Options:
PHI Level: Use the drop down to set the Protected Health Information (PHI) level of data in this field. Note that setting a field as containing PHI does not automatically provide any protection of that data.
Exclude from "Participant Date Shifting" on export/publication: (Date fields only) If the option to shift/randomize participant date information is selected during study folder export or publishing of a study, do not shift the dates in this field. Use caution when protecting PHI to ensure you do not exempt fields you intend to shift.
Make this field available as a measure: Check the box for fields that contain data to be used for charting and other analysis. These are typically numeric results. Learn more about using Measures and Dimensions for analysis.
Make this field available as a dimension: (Not available for Date fields) Check the box for fields to be used as 'categories' in a chart. Dimensions define logical groupings of measures and are typically non-numerical, but may include numeric type fields. Learn more about using Measures and Dimensions for analysis.
Make this field a recommended variable: Check the box to indicate that this is an important variable. These variables will be displayed as recommended when creating new participant reports.
Track reason for missing data values: Check this box to enable the field to hold special values to indicate data that has failed review or was originally missing. Administrators can set custom Missing Value indicators at the site and folder levels. Learn more about using Missing Value Indicators.
Require all values to be unique: Check this box to add a uniqueness constraint to this field. You can only set this property if the data currently in the field is unique. Supported for Lists, Datasets, Sample Types, and Data Classes.
View Fields in Summary Mode
In the upper right of the field editor, the Mode selection lets you choose:
Detail: You can open individual field panels to edit details.
Summary: A summary of set values is shown; limited editing is possible in this mode.
In Summary Mode, you see a grid of fields and properties, and a simplified interface. Scroll for more columns. Instead of having to expand panels to see things like whether there is a URL or formatting associated with a given field, the summary grid makes it easier to scan and search large sets of fields at once.You can add new fields, delete selected fields, and export fields while in summary mode.
Export Sets of Fields (Domains)
Once you have defined a set of fields (domain) that you want to be able to save or reuse, you can export some or all of the fields by clicking (Export). If you have selected a subset of fields, only the selected fields will be exported. If you have not selected fields (or have selected all fields) all fields will be included in the export.A Fields_*.fields.json file describing your fields as a set of key/value pairs will be downloaded. All properties that can be set for a field in the user interface will be included in the exported file contents. For example, the basic text field named "FirstField" we show above looks like:
You can save this file to import a matching set of fields elsewhere, or edit it offline to reimport an adjusted set of fields.For example, you might use this process to create a dataset in which all fields were marked as measures. Instead of having to open each field's advanced properties in the user interface, you could find and replace the "measure" setting in the JSON file and then create the dataset with all fields set as measures.
Note that importing fields from a JSON file is only supported when creating a new set of fields. You cannot apply property settings to existing data with this process.
The set of fields that make up a data structure like a list, dataset, assay, etc. can be edited using the Field Editor interface. You will find instructions about using the field editor and using properties, options, and settings common to all field types in the topic: Field EditorThis topic outlines the field formatting and properties specific to each data type.
Fields come in different types, each intended to hold a different kind of data. Once defined, there are only limited ways you can change a field's type, based on the ability to convert existing data to the new type. To change a field type, you may need to delete and recreate it, reimporting any data.The following table show which fields are available in which kind of table/data structure.
Notice that Datasets do not support Attachment fields. For a workaround technique, see Linking Data Records to Image Files.
The SMILES field type is only available in the Compounds data class when using LabKey Biologics LIMS.
Changes Allowed by Field Type
Once defined, there are only limited ways you can change a field's data type, based on the ability to safely convert existing data to the new type. This table lists the changes supported (not all types are available in all data structures and contexts):
The basic properties and validation available for different types of fields are covered in the main field editor topic. Details for specific types of fields are covered here.
Text, Multi-Line Text, and Flag Options
Fields of type Text, Multi-Line Text, and Flag have the same set of properties and formats available:
Text Options/Multi-line Text Field Options/Flag Options: Maximum Text Length. Sets the maximum character count for the field. Options are:
When setting the Maximum Text Link, consider that when you make a field UNLIMITED, it is stored in the database as 'text' instead of the more common 'varchar(N)'. When the length of a string exceeds a certain amount, the text is stored out of the row itself. In this way, you get the illusion of an infinite capacity. Because of the differences between 'text' and 'varchar' columns, these are some good general guidelines:
UNLIMITED is good if:
You need to store large text (like, a paragraph or more)
You don’t need to index the column
You will not join on the contents of this column
Examples: blog comments, wiki pages, etc.
No longer than... (i.e. not-UNLIMITED) is good if:
You're storing smaller strings
You'll want them indexed, i.e. you plan to search on string values
You want to do a sql select or join on the text in this column
Examples would be usernames, filenames, etc.
Text Choice Options
A Text Choice field lets you define a set of values that will be presented to the user as a dropdown list. This is similar to a lookup field, but does not require a separate list created outside the field editor. Click Add Values to open a panel where you can add the choices for this field.
Text Choice Options:
Add and manage the set of Drop-down values offered for this field.
Note that because administrators can always see all the values offered for a Text Choice field, they are not a good choice for storing PHI or other sensitive information. Consider instead using a lookup to a protected list.
Boolean Options
Boolean Field Options: Format for Boolean Values: Use boolean formatting to specify the text to show when a value is true and false. Text can optionally be shown for null values. For example, "Yes;No;Blank" would output "Yes" if the value is true, "No" if false, and "Blank" for a null value.
Integer and Decimal fields share similar number formatting options, shown below. When considering which type to choose, keep in mind the following behavior:
Integer: A 4-byte signed integer that can hold values ranging -2,147,483,648 to +2,147,483,647.
Decimal (Floating Point): An 8-byte double precision floating point number that can hold very large and very small values. Values can range approximately 1E-307 to 1E+308 with a precision of at least 15 digits. As with most standard floating point representations, some values cannot be converted exactly and are stored as approximations. It is often helpful to set on Decimal fields a display format that specifies a fixed or maximum number of decimal places to avoid displaying approximate values.
Both Integer and Decimal columns have these formatting options available:
Integer Options/Decimal Options:
Format for Numbers: To control how a number value is displayed, provide a string format compatible with the Java class DecimalFormat. Learn more about using Number formats in LabKey.
Three different field types are available for many types of data structure, letting you choose how best to represent the data needed.
Date Time: Both date and time are included in the field. Fields of this type can be changed to either "Date-only" or "Time-only" fields, though this change will drop the data in the other part of the stored value.
Date: Only the date is included. Fields of this type can be changed to be "Date Time" fields.
Time: Only the time portion is represented. Fields of this type cannot be changed to be either "Date" or "Date Time" fields.
Date and Time Options:
Use Default: Check the box to use the default format set for this folder.
Format for Date Times: Uncheck the default box to enable a drop down where you can select the specific format to use for dates/times in this field. Learn more about using Date and Time formats in LabKey.
Conditional Formatting and Validation Options: Conditional formats are available for all three types of date time fields. Range validators are available for "Date" and "Date Time" but not for "Time" fields.
Calculation (Premium Feature)
Calculation fields are available with the Professional and Enterprise Editions of LabKey Server, the Professional Edition of Sample Manager, LabKey LIMS, and Biologics LIMS.A calculation field lets you include SQL expressions using values in other fields in the same row to provide calculated values. The Expression provided must be valid LabKey SQL and can use the default system fields, custom fields, constants, operators, and functions. To use field names containing special characters in a calculated field, surround the name with double quotes. String constants use single quotes. Examples:
Operation
Example
Addition
numericField1 + numericField2
Subtraction
numericField1 - numericField2
Multiplication
numericField1 * numericField2
Division by value known never to be zero
numericField1 / nonZeroField1
Division by value that might be zero
CASE WHEN numericField2 <> 0 THEN (numericField1 / numericField2 * 100) ELSE NULL END
Subtraction of dates/datetimes (ex: difference in days)
CASE WHEN FreezeThawCount < 2 THEN 'Viable' ELSE 'Questionable' END
Conditional calculation based on a text match
CASE WHEN ColorField = 'Blue' THEN 'Abnormal' ELSE 'Normal' END
Text value for every row (ex: to use with a URL property)
'clickMe'
Text concatenation (use fields and/or strings)
City || ', ' || State
Addition when field name includes special characters
"Numeric Field Name" + "Field/Name & More"
Once you've provided the expression, use Click to validate to confirm that your expression is valid.
The data type of your expression will be calculated. In some cases, you may want to use casts to influence this type determination. For example, if you divide two integer values, the result will also be of type integer and yield unexpected and apparently "truncated" results.Once the type of the calculation is determined, you'll see additional formatting options for the calculated type of the field. Shown above, date and time options.
File and Attachment fields are only available in specific scenarios, and have display, thumbnail, and storage differences. For both types of field, you can configure the behavior when a link is clicked, so that the file or attachment either downloads or is shown in the browser.
If you are using a pipeline override, note that it will override any file root setting, so the "file repository" in this case will be under your pipeline override location.
Attachment
The Attachment field type is similar to File, but only available for lists and data classes (including Sources for samples).
This type allows you to attach documents to individual records in a list or data class.
For instance, an image could be associated with a given row of data in an attachment field, and would show an inline thumbnail.
The attachment is not uploaded into the file repository, but is stored as a BLOB field in the database.
By default, the maximum attachment size is 50MB, but this can be changed in the Admin Console using the setting Maximum file size, in bytes, to allow in database BLOBs. See Site Settings.
Learn about using File and Attachment fields in Sample Manager and LabKey Biologics in this topic:
When a field of type File or Attachment is an image, such as a .png or .jpg file, the cell in the data grid will display a thumbnail of the image. Hovering reveals a larger version.When you export a grid containing these inline images to Excel, the thumbnails remain associated with the cell itself.
Bulk Import into the File Field Type
You can bulk import data into the File field type in LabKey Server, provided that the files/images are already uploaded to the File Repository, or the pipeline override location if one is set. For example suppose you already have a set of images in the File Repository, as shown below.You can load these images into a File field, if you refer to the images by their full server path in the File Repository. For example, the following shows how an assay upload might refer to these images by their full server path:
This field type is only available for Study datasets and for Sample Types that will be linked to study data. The Subject/Participant ID is a concept URI, containing metadata about the field. It is used in assay and study folders to identify the subject ID field. There is no special built-in behavior associated with this type. It is treated as a string field, without the formatting options available for text fields.
You can populate a field with data via lookup into another table. This is similar to the text choice field, but offers additional flexibility and longer lists of options.Open the details panel and select the folder, schema, and table where the data values will be found. Users adding data for this field will see a dropdown populated with that list of values. Typing ahead will scroll the list to the matching value. When the number of available values exceeds 10,000, the field will be shown as a text entry field.Use the checkbox to control whether the user entered value will need to match an existing value in the lookup target.
Lookup Definition Options:
Select the Target Folder, Schema, and Table from which to look up the value. Once selected, the value will appear in the top row of the field description as a direct link to the looked-up table.
Lookup Validator: Ensure Value Exists in Lookup Target. Check the box to require that any value is present in the lookup's target table or query.
Note that lookups into lists with auto-incrementing keys may not export/import properly because the rowIds are likely to be different in every database.Learn more about Lookup fields in this topic:
Integration of Sample Types with Study data is supported using Visit Date, Visit ID, and Visit Label field types provide time alignment, in conjunction with a Subject/Participant field providing participant alignment. Learn more in this topic: Link Sample Data to Study
A Text Choice field lets you define a set of values that will be presented to the user as a dropdown list. This is similar to a lookup field, but does not require a separate list created outside the field editor. Using such controlled vocabularies can both simplify user entry and ensure more consistent data.This topic covers the specifics of using the Text Choice Options for the field. Details about the linking and conditional formatting of a text choice field are provided in the shared field property documentation.
Open the field editor for your data structure, and locate or add a field of type Text Choice.Click Add Values to open a panel where you can add the choices for this field. Choices may be multi-word.Enter each value on a new line and click Apply.
Manage Text Choices
Once a text choice field contains a set of values, the field summary panel will show at least the first few values for quick reference.Expand the field details to see and edit this list.
Add Values lets you add more options.
Delete a value by selecting, then clicking Delete.
Click Apply to save your change to this value. You will see a message indicating updates to existing rows will be made.
Click Save for the data structure when finished editing text choices.
In-Use and Locked Text Choices
Any values that are already in use in the data will be marked with a icon indicating that they cannot be deleted, but the text of the value may be edited.Values that are in-use for any read-only data (i.e. assay data that cannot be edited) will be marked with a icon indicating that they cannot be deleted or edited.
PHI Considerations
When a text choice field is marked as PHI, the usual export and publish study options will be applied to data in the field. However, users who are able to access the field editor will be able to see all of the text choices available, though without any row associations. If you are also using the compliance features letting you hide PHI from unauthorized users, this could create unintended visibility of values that should be obscured.For more complete protection of protected health information, use a lookup to a protected list instead.
Use a Text Choice Field
When a user inserts into or edits the value of the Text Choice field, they can select one of the choices, or leave it blank if it is not marked as a required field. Typing ahead will narrow longer lists of choices. Entering free text is not supported. New choices can only be added within the field definition.
Change Between Text and Text Choice Field Types
Fields of the Text and Text Choice types can be switched to the other type without loss of data. Such changes have a few behaviors of note:
If you change a Text Choice field to Text, you will lose any values on the list of options for the field that are not in use.
If you change a Text field to Text Choice, all distinct values in that column will be added to the values list for the field. This option creates a handy shortcut for when you are creating new data structures including text choice fields.
Importing Text Choices: A Shortcut
If you have an existing spreadsheet of data (such as for a list or dataset) and want a given field to become a text choice field, you have two options.
The more difficult is to create the data structure with the field of type Text Choice, and then copy and paste all the distinct values from the dataset into the drop-down options for the field, then import your data.
An easier option is to first create the structure and import the data selecting Text as the type for the field you want to be a dropdown. Once imported, edit the data structure to change the type of that field to Text Choice. All distinct values from the column will be added to the list of value options for you.
Setting the URL property of a field in a data grid turns the display value into a link to other content. The URL property setting is the target address of the link. The URL property supports different options for defining relative or absolute links, and also supports using values from other fields in constructing the URL.
Several link format types for URL property are supported: local, relative, and full path links on the server as well as external links. Learn more about the context, controller, action, and path elements of LabKey URLs in this topic: LabKey URLs.
Local Links
To link to content in the current LabKey folder, use the controller and action name, but no path information.
<controller>-<action>
For example, to view a page in a local wiki:
wiki-page.view?name=${PageName}
A key advantage of this format is that the list or query containing the URL property can be moved or copied to another folder with the target wiki page, in this case, and it will still continue to work correctly.You can optionally prepend a / (slash) or ./ (dot-slash), but they are not necessary. You could also choose to format with the controller and action separated by a / (slash). The following format is equivalent to the above:
./wiki/page.view?name=${PageName}
Relative Links
To point to resources in subfolders of the current folder, prepend the local link format with path information, using the . (dot) for the current location:
./<subfoldername/><controller>-<action>
For example, to link to a page in a subfolder, use:
./${SubFolder}/wiki-page.view?name=${PageName}
You can also use .. (two dots) to link to the parent folder, enabling path syntax like "../siblingfolder/resource to refer to a sibling folder. Use caution when creating complex relative URL paths as it makes references more difficult to follow in the case of moving resources. Consider a full path to be more clear when linking to a distant location.
Full Path Links on the Same Server
A full path link points to a resource on the current LabKey Server, useful for:
linking commmon resources in shared team folders
when the URL is a WebDAV link to a file that has been uploaded to the current server
The local path begins with a / (forward slash).For example, if a page is in a subfolder of "myresources" under the home folder, the path might be:
To link to a resource on an external server or any website, include the full URL link. If the external location is another labkey server, the same path, controller, and action path information will apply:
http://server/path/page.html?id=${Param}
Substitution Syntax
If you would like to have a data grid contain a link including an element from elsewhere in the row, the ${ } substitution syntax may be used. This syntax inserts a named field's value into the URL. For example, in a set of experimental data where one column contains a Gene Symbol, a researcher might wish to quickly compare her results with the information in The Gene Ontology. Generating a URL for the GO website with the Gene Symbol as a parameter will give the researcher an efficient way to "click through" from any row to the correct gene.An example URL (in this case for the BRCA gene) might look like:http://amigo.geneontology.org/amigo/search/ontology?q=brcaSince the search_query parameter value is the only part of the URL that changes in each row of the table, the researcher can set the URL property on the GeneSymbol field to use a substitution marker like this:http://amigo.geneontology.org/amigo/search/ontology?q=${GeneSymbol} Once defined, the researcher would simply click on "BRCA" in the correct column to be linked to the URL with the search_query parameter applied.Multiple such substitution markers can be used in a single URL property string, and the field referenced by the markers can be any field within the query.Substitutions are allowed in any part of the URL, either in the main path, or in the query string. For example, here are two different formats for creating links to an article in wikipedia, here using a "CompanyName" field value:
The following substitutions markers are built-in and available for any query/dataset. They help you determine the context of the current query.
Marker
Description
Example Value
${schemaName}
The schema where the current query lives.
study
${schemaPath}
The schema path of the current query.
assay.General.MyAssayDesign
${queryName}
The name of the current query
Physical Exam
${dataRegionName}
The data region for the current query.
Dataset
${containerPath}
The LabKey Server folder path, starting with the project
/home/myfolderpath
${contextPath}
The Tomcat context path
/labkey
${selectionKey}
Unique string used by selection APIs as a key when storing or retrieving the selected items for a grid
$study$Physical Exam$$Dataset
Link Display Text
The display text of the link created from a URL property is just the value of the current record in the field which contains the URL property. So in the Gene Ontology example, since the URL property is defined on the Gene_Symbol field, the gene symbol serves as both the text of the link and the value of the search_query parameter in the link address. In many cases you may want to have a constant display text for the link on every row. This text could indicate where the link goes, which would be especially useful if you want multiple such links on each row.In the example above, suppose the researcher wants to be able to look up the gene symbol in both Gene Ontology and EntrezGene. Rather than defining the URL Property on the Gene_Symbol field itself, it would be easier to understand if two new fields were added to the query, with the value in the fields being the same for every record, namely "[GO]" and "[Entrez]". Then set the URL property on these two new fields tofor the GOlink field:
The resulting query grid will look like:Note that if the two new columns are added to the underlying list, dataset, or schema table directly, the link text values would need to be entered for every existing record. Changing the link text would also be tedious. A better approach is to wrap the list in a query that adds the two fields as constant expressions. For this example, the query might look like:
Then in the Edit Metadata page of the Schema Browser, set the URL properties on these query expression fields:
URL Encoding Options
You can specify the type of URL encoding for a substitution marker, in case the default behavior doesn't work for the URLs needed. This flexibility makes it possible to have one column display the text and a second column can contain the entire href value, or only a part of the href.
The fields referenced by the ${ } substitution markers might contain any sort of text, including special characters such as question marks, equal signs, and ampersands. If these values are copied straight into the link address, the resulting address would be interpreted incorrectly. To avoid this problem, LabKey Server encodes text values before copying them into the URL. In encoding, characters such as ? are replaced by their character code %3F. By default, LabKey encodes all special character values except '/' from substitution markers. If you know that a field referenced by a substitution marker needs no encoding (because it has already been encoded, perhaps) or needs different encoding rules, inside the ${ } syntax, you can specify encoding options as described in the topic String Expression Format Functions.
Links Without the URL Property
If the data field value contains an entire url starting with an address type designator (http:, https:, etc), then the field value is displayed as a link with the entire value as both the address and the display text. This special case could be useful for queries where the query author could create a URL as an expression column. There is no control over the display text when creating URLs this way.
Linking To Other Tables
To link two tables, so that records in one table link to filtered views of the other, start with a filtered grid view of the target table, filtering on the target fields of interest. For example, the following URL filters on the fields "WellLocation" and "WellType":
Parameterize by adding substitution markers within the filter. For example, assume that source and target tables have identical field names, "WellLocation" and "WellType":
Query Metadata: Examples: Includes having a URL open in a new tab and displaying a thumbnail image for your URL link.
For an example of UI usage, see: Step 3: Add a URL Property and click here to see an interactive example. Hover over a link in the Department column to see the URL, click to view a list filtered to display the "technicians" in that department.For examples of SQL metadata XML usage, see: JavaScript API Demo Summary Report and the JavaScript API Tutorial.
Conditional formats change how data is displayed depending on the value of the data. For example, if temperature goes above a certain value, you can highlight those values using orange. If the value is below a certain level those could be blue. Bold, italic, and strikethrough text can also be used.Conditional formats are available in LabKey Server, LabKey LIMS, and Biologics LIMS. They are defined as properties of fields using the Field Editor.
Specify a Conditional Format
To specify a conditional format, open the field editor, and click to expand the field of interest. Under Create Conditional Format Criteria, click Add Format.In the popup, identify the condition(s) under which you want the conditional format applied. Specifying a condition is similar to specifying a filter. You need to include a First Condition. If you specify a second one, both will be AND-ed together to determine whether a single conditional format is displayed.Only the value that is being formatted is available for the condition checks. That is, you cannot use the value in column A to apply a conditional format to column B.Next, you can specify Display Options, meaning how the field should be formatted when that condition is met.Display options are:
Bold
Italic
Strikethrough
Colors: Select Text and/or Fill colors. Click a block to choose it, or type to enter a hex value or RGB values. You'll see a box of preview text on the right.
Click Apply to close the popup, then Save.
When you view the table, you'll see your formatting applied.
Multiple Conditional Formats
Multiple conditional formats are supported in a single column. Before applying the format, you can click Add Formatting to add another. Once you have saved an active format, use Edit Formats to reopen the popup and click Add Formatting to specify another conditional format. This additional condition can have a different type of display applied.Each format you define will be in a panel within the popup and can be edited separately.If a value fulfills multiple conditions, then the first condition satisfied is applied, and conditions lower on the list are ignored.For example, suppose you have specified two conditional formats on one field:
If the value is 40 degrees or greater, then display in bold text.
If the value is 38 degrees or greater, then display in italic text.
Although the value 40 fulfills both conditions, only the first condition to apply is considered, resulting in bold display.
Example: Conditional Formats for Temperature
In the following example, values out of the normal human body temperature range are highlighted with color if too high and shown in italics if too low. In this example, we use the Physical Exam dataset that is included with the importable example study.
In a grid view of the Physical Exam dataset, click Manage.
Click Edit Definition.
Select a field (such as temperature in this example), expand it, and click Add Format under "Create Conditional Format Criteria".
For First Condition, choose "Is Greater Than", enter 37.8.
Check Bold.
From the Fill Color drop down, choose orange for this example.
This format option is shown above.
Click Add Formatting in the popup before clicking Apply.
For First Condition of this second format, choose "Is Less Than", enter 36.1.
Check the box for Italics.
Choose a blue Fill Color.
Click Apply, then Save.
Click View Data to return to the data grid.
Now temperature values above 37.8 degrees are in bold on orange cells and those below 36.1 are displayed in italics with a blue background.When you hover over a formatted value in the LabKey Server interface, a pop up dialog will appear explaining the rule behind the format. Note that these popups are not available in the LIMS products.
View Conditional Formats in LIMS Products
In LabKey LIMS and Biologics LIMS, you use the same mechanism to set formatting as described above. Instead of filling the entire cell, background colors are displayed as a lozenge as shown here.The conditional formatting is also shown in individual details views for an entity:
Escape carrage return, linefeed, and <>"' characters and surround with a single quotes
${field:jsString}
urlEncode
path
string
URL encode each path part preserving path separator
${field:urlEncode}
String
join(string)
collection
Combine a collection of values together separated by the string argument
${field:join('/'):encodeURI}
prefix(string)
string, collection
Prepend a string argument if the value is non-null and non-empty
${field:prefix('-')}
suffix(string)
string, collection
Append a string argument if the value is non-null and non-empty
${field:suffix('-')}
trim
string
Remove any leading or trailing whitespace
${field:trim}
Date
date(string)
date
Format a date using a format string or one of the constants from Java's DateTimeFormatter. If no format value is provided, the default format is 'BASIC_ISO_DATE'
Drop all items from the collection except the last
${field:last:suffix('!')}
Examples
Function
Applied to...
Result
${Column1:defaultValue('MissingValue')}
null
MissingValue
${Array1:join('/')}
[apple, orange, pear]
apple/orange/pear
${Array1:first}
[apple, orange, pear]
apple
${Array1:first:defaultValue('X')}
[(null), orange, pear]
X
Date & Number Display Formats
LabKey Server provides flexible display formatting for dates, times, datetimes, and numbers, so you can control how these values are shown to users. Set up formats that apply to the entire site, an entire project, a single folder, or even just to an individual field in one table.
You can customize how dates, times, datetimes, and numbers are displayed on a field-by-field basis, or set these formats on a folder-level, project-level or site-wide basis. The server decides which format to use for a particular field by looking first at the properties for that field. If no display format is set at the field-level, it checks the container tree, starting with the folder then up the folder hierarchy to the site level. In detail, decision process goes as follows:
The server checks to see if there is a field-level format set on the field itself. If it finds a field-specific format, it uses that format. If no format is found, it looks to the folder-level format. (To set a field-specific format, see Set Formats on a Per-Field Basis.)
If a folder-level format is found, it uses that format. If no folder-level format is found, it looks in the parent folder, then that parent's parent folder, etc. until the project level is reached and it looks there. (To set a folder-level default format, see Set Folder Display Formats)
If a project-level format is found, it uses that format. If no project-level format is found, it looks to the site-level format. (To set a project-level default format, see Set Project Display Formats.)
To set the site-level format, see Set Formats Globally (Site-Level). Note that if no site-level format is specified, the server will default to these formats:
Date format: yyyy-MM-dd
Time format: HH:mm
Date, time, and date-time formats are selected from a set of built in options. A standard Java format string specifies how numbers are displayed.
Set Site-Wide Display Formats
An admin can set formats at the site level by managing look and feel settings.
Select > Site > Admin Console.
Under Configuration, click Look and Feel Settings.
Scroll down to Customize date, time, and number display formats.
Select desired formats for date, date-time, and time-only fields.
Check the "Inherited" checkbox to use the project-wide format, or uncheck to set folder-specific formats:
Select desired formats for date, date-time, and time-only fields.
Enter desired format for number fields.
Click Save.
Set Formats on a Per-Field Basis
To do this, you edit the properties of the field.
Open the data fields for editing; the method depends on the type. See: Field Editor.
Expand the field you want to edit.
Date, time, and datetime fields include a Use Default checkbox, and if unchecked you can use the Format for Date Times selectors to choose among supported date and time formats just for this field.
Number fields include a Format for Numbers box. Enter the desired format string.
Click Save.
Date, Time, and DateTime Display Formats
Date, Time, and DateTime display formats are selected from a set of standard options, giving you flexibility for how users will see these values. DateTime fields combine one of each format, with the option of choosing "<none>" as the Time portion.Date formats available:
Format Selected
Display Result
yyyy-MM-dd
2024-08-14
yyyy-MMM-dd
2024-Aug-14
yyyy-MM
2024-08
dd-MM-yyyy
14-08-2024
dd-MMM-yyyy
14-Aug-2024
dd-MMM-yy
14-Aug-24
ddMMMyyyy
14Aug2024
ddMMMyy
14Aug24
MM/dd/yyyy
08/14/2024
MM-dd-yyyy
08-14-2024
MMMM dd yyyy
August 14 2024
Time formats available:
Format Selected
Display Results
HH:mm:ss
13:45:15
HH:mm
13:45
HH:mm:ss.SSS
13:45:15.000
hh:mm a
01:45 PM
Number Format Strings
Format strings for Number (Double) fields must be compatible with the format that the java class DecimalFormat accepts. A valid DecimalFormat is a pattern specifying a prefix, numeric part, and suffix. For more information see the Java documentation. The following table has an abbreviated guide to pattern symbols:
Symbol
Location
Localized?
Meaning
0
Number
Yes
Digit
#
Number
Yes
Digit, zero shows as absent
.
Number
Yes
Decimal separator or monetary decimal separator
-
Number
Yes
Minus sign
,
Number
Yes
Grouping separator
E
Number
Yes
Exponent for scientific notation, for example: 0.#E0
Examples
The following examples apply to Number (Double) fields.
By combining data from multiple tables in one grid view, you can create integrated grids and visualizations with no duplication of data. Lookup Columns give you the ability to link data from two different tables by reference. Another name for this type of connection is a "foreign key".Once a lookup column pulls in data from another table, you can then display values from any column in that target (or "looked up") table. You can also take advantage of a lookup to simplify user data entry by constraining entered values for a column to a fixed set of values in another list.
Suppose you want to display values from the Languages list, such as the translator information, alongside other data from the Demographics dataset. You would add a lookup column to the Demographics dataset that used values from the Languages list.To join these tables, an administrator adds a lookup column to the Demographics dataset definition using this interface:
Go to the dataset or list where you want to show the data from the other source. Here, Demographics.
Click Manage, then Edit Definition.
Click Fields.
Expand the field you want to be a lookup (here "language").
If the field you want doesn't already exist, click Add Field to add it.
From the dropdown under Data Type, select Lookup.
In the Lookup Definition Options, select the Target Folder, Schema, and Table to use. For example, the lists schema and the Languages table, as shown below.
Scroll down and click Save.
In either the expanded or collapsed view, you can click the name of the target of the lookup to open it directly.
The lookup column is now available for grid views and SQL queries on the Demographics table.
This can also be accomplished using a SQL query, as described in Lookups: SQL Syntax.
Create a Joined Grid View
Once you have connected tables with a lookup, you can create joined grids merging information from any of the columns of both tables. For example, a grid showing which translators are needed for each cohort would make it possible to schedule their time efficiently. Note that the original "Language" column itself need not be shown.
Go to the "Demographics" data grid.
Select (Grid Views) > Customize Grid.
Click the to expand the lookup column and see the fields it contains. (It will become a as shown below.)
Select the fields you want to display using checkboxes.
Save the grid view.
Learn more about customizing grid views in this topic:
For lookup fields where the target table has an integer primary key, the server will use the first text field it encounters as the default display column. For example, suppose the Language field is an integer lookup to the Languages table, as below.In this case, the server uses Language Name as the default display field because it is the first text field it finds in the looked up table. You can see this in the details of the lookup column shown in the example above. The names "English", etc, are displayed though the lookup is to an integer key.
Display Alternate Fields
To display other fields from the looked up table, go to (Grid Views) > Customize View, expand the lookup column, and select the fields you want to display.You can also use query metadata to achieve the same result: see Query Metadata: Examples
Validating Lookups: Enforcing Lookup Values on Import
When you are importing data into a table that includes a lookup column, you can have the system enforce the lookup values, i.e. any imported values must appear in the target table. An error will be displayed whenever you attempt to import a value that is not in the lookup's target table.To set up enforcement:
Go to the field editor of the dataset or list with the lookup column.
Select the lookup column in the Fields section.
Expand it.
Under Lookup Validator, check the box for Ensure Value Exists in Lookup Target.
Click Save.
Note that pre-existing data is not retroactively validated by turning on the lookup validator. To ensure pre-existing data conforms to the values in the lookup target table, either review entries by hand or re-import to confirm values.
Suppress Linking
By default the display value in a lookup field links to the target table. To suppress the links, and instead display the value/data as bare text, edit the XML metadata on the target table. For example, if you are looking up to MyList, add <tableUrl></tableUrl> to its metadata, as follows. This will suppress linking on any table that has a lookup into MyList.
A related option is to use a SQL annotation directly in the query to not "follow" the lookup and display the column as if there were no foreign key defined on it. Depending on the display value of the lookup field, you may see a different value than if you used the above suppression of the link. Learn more here: Query Metadata.
Related Topics
Filtered Lookups - Control the values shown in a lookup dropdown.
In many research applications, Protected Health Information (PHI) is collected and available to authorized users, but must not be shared with unauthorized users. This topic covers how to mark columns (fields) as different levels of PHI and how you can use these markers to control access to information.Administrators can mark columns as either Restricted PHI, Full PHI, Limited PHI, or Not PHI. Simply marking fields with a particular PHI level does not restrict access to these fields, and in some cases the setting has special behavior as detailed below.
Note that while you can set the PHI level for assay data fields, assays do not support using PHI levels to restrict access to specific fields in the ways described here. Control of access to assay data should be accomplished by using folder permissions and only selectively copying non-PHI data to studies.
Click the Fields section if it is not already open.
Expand the field you want to mark.
Click Advanced Settings.
From the PHI Level dropdown, select the level at which to mark the field. Shown here, the "country" field is being marked as "Limited PHI".
Click Apply.
Continue to mark other fields in the same data structure as needed.
Click Save when finished.
Developers can also use XML metadata to mark a field as containing PHI.Once your fields are marked, you can use this information to control export and publication of studies and folders.
Some Columns Cannot be Marked as PHI
There are certain fields which cannot be annotated as PHI because to do so would interfere with system actions like study alignment and usability. For example, the ParticipantID in a study cannot be marked as containing PHI.Instead, in order to protect Participant identifiers that are considered PHI, you can:
Create new participantIDs unrelated to the originals and provide an additional text column for the "PHI" identifiers, marking that other column as PHI.
When you export a folder or study, you can select the level(s) of PHI to include. By default, all columns are included, so to exclude any columns, you must make a selection as follows.
Select > Folder > Management.
Click the Export tab.
Select the objects you want to export.
Under Options, choose which levels of PHI you want to include.
Note that assay data does not support PHI field settings and if selected, all data will be included if selected, regardless of the PHI level you select here.
Uncheck the Include PHI Columns box to exclude all columns marked as PHI.
Click Export.
The exported archive can now be shared with users who can have access to the selected level of PHI.This same option can be used when creating a new folder from a template, allowing you to create a new similar folder without any PHI columns.
Publish Study without PHI
When you publish a study, you create a copy in a new folder, using a wizard to select the desired study components. On the Publish Options panel, you can select the PHI you want to include. The default is to publish with all columns.
In a study, click the Manage tab.
Click Publish Study.
Complete the study wizard selecting the desired components.
On the Publish Options panel, under Include PHI Columns, select the desired levels to include.
Click Finish.
The published study folder will only contain the columns at the level(s) you included.
Issues List and Limited PHI
An issue tracking list ordinarily shows all fields to all users. If you would like to have certain fields only available to users with permission to insert or update issues (Submitter, Editor, or Admins), you can set fields to "Limited PHI".For example, a development bug tracker could have a "Limited PHI" field indicating the estimated work needed to resolve it. This field would then be hidden from readers of the list of known issues but visible to the group planning to fix them.Learn more about customizing issue trackers in this topic: Issue Tracker: Administration
Use PHI Levels to Control UI Visibility of Data
Premium Features AvailableSubscribers to the Enterprise Edition of LabKey Server can use PHI levels to control display of columns in the user interface. Learn more in this topic:
Note that if your data uses any text choice fields, administrators and data structure editors will be able to see all values available within the field editor, making this a poor field choice for sensitive information.
Data grids display data from various data structures as a table of columns and rows. LabKey Server provides sophisticated tools for working with data grids, including sorting, filtering, reporting, and exporting.Take a quick tour of the basic data grid features here: Data Grid Tour.
Data in LabKey Server is viewed in grids. The underlying data table, list, or dataset stored in the database contains more information than is shown in any given grid view. This topic will help you learn the basics of using grids in LabKey to view and work with any kind of tabular data.
Note: If you are using Sample Manager, LabKey LIMS, or Biologics LIMS, there are some variations in the appearance and features available for data grids. Learn more in this topic: Grid Basics in Sample Manager and LIMS Products
Anatomy of a Data Grid
The following image shows a typical data grid.
Data grid title: The name of the data grid.
QC State filter: Within a LabKey study, when one or more dataset QC states are defined, the button bar will include a menu for filtering among them. The current status of that filter is shown above the grid.
Folder location: The folder location. Click to navigate to the home page of the folder.
Grid view indicator: Shows which view, or perspective, on the data is being shown. Custom views are created to show a joined set of tables or highlight particular columns in the data. Every dataset has a "default" view that can also be customized if desired.
Filter bar: When filters are applied, they appear here. Click the X to remove a filter. When more than one is present, a Clear all button is also shown.
Button bar: Shows the different tools that can be applied to your data. A triangle indicates a menu of options.
Column headers: Click a column header for a list of options.
Data records: Displays the data as a 2-dimensional table of rows and columns.
Page through data: Control how much data is shown per page and step through pages.
The button bar tools available may vary with different types of data grid. Study datasets can provide additional functionality, such as filtering by cohort, that are not available to lists. Assays and proteomics data grids provide many additional features.Hover over icon buttons to see a tooltip with the text name of the option. Common buttons and menus include:
(Filter): Click to open a panel for filtering by participant groups and cohorts.
(Grid Views): Pull-down menu for creating, selecting, and managing various grid views of this data.
(Charts/Reports): Pull-down menu for creating and selecting charts and reports.
(Export): Export the data grid as a spreadsheet, text, or in various scripting formats.
(Insert Data): Insert a new single row, or bulk import data.
(Delete): Delete one or more rows selected by checkbox. This option is disabled when no rows are selected.
Design/Manage: with appropriate permissions, change the structure and behavior of the given list ("Design") or dataset ("Manage").
Groups: Select which cohorts, participant groups, or treatment groups to show. Also includes options to manage cohorts and create new participant groups.
Print: Generate a printable version of the dataset in your browser.
Page Through Data
In the upper right corner, you will see how your grid is currently paginated. In the above example, there are 484 rows in the dataset and they are shown 100 to a "page", so the first page is rows 1-100 of 484. To step through pages, in this case 100 records at a time, click the arrow buttons. To adjust paging, hover over the row count message (here "1-100 of 484") and it will become a button. Click to open the menu:
Show First/Show Last: Jump to the first or last page of data.
Show All: Show all rows in one page.
Click Paging to see the currently selected number of rows per page. Notice the > caret indicating there is a submenu to open. On the paging submenu, click another option to change pagination.
Examples
The following links show different views of a single data grid:
LabKey provides a variety of methods for importing data into a data grid. Depending on the type and complexity of the data, you must first identify the type of data structure in which your data will be stored. Each data structure has a different specific process for designing a schema and then importing data.
Note: When data is imported as a TSV, CSV, or text file, it is parsed using UTF-8 character encoding.
Sort Data
This page explains how to sort data in a table or grid. The basic sort operations are only applied for the current viewer and session; they are not persisted by default, but can be saved as part of a custom grid view.
To sort data in a grid view, click on the header for a column name. Choose either:
Sort Ascending: Show the rows with lowest values (or first values alphabetically) for this column at the top of the grid.
Sort Descending: Show the rows with the highest values at the top.
Once you have sorted your dataset using a particular column, a triangle icon will appear in the column header: for an ascending sort or for a descending sort.You can sort a grid view using multiple columns at a time as shown above. Sorts are applied in the order they are added, so the most recent sort will show as the primary way the grid is sorted. For example, to sort by date, with same-date results sorted by temperature, you would first sort on temperature, then on date.Note that LabKey sorting is case-sensitive.
Clear Sorts
To remove a sort on an individual column, click the column caption and select Clear Sort.
Advanced: Understand Sorting URLs
The sort specifications are included on the page URL. You can modify the URL directly to change the sorted columns, the order in which they are sorted, and the direction of the sort. For example, the following URL sorts the Physical Exam grid first by ascending ParticipantId, and then by descending Temp_C:
Note that the minus ('-') sign in front of the Temp_C column indicates that the sort on that column is performed in descending order. No sign is required for an ascending sort, but it is acceptable to explicitly specify the plus ('+') sign in the URL.The %2C hexadecimal code that separates the column names represents the URL encoding symbol for a comma.
You can filter data displayed in a grid to reduce the amount of data shown, or to exclude data that you do not wish to see. By default, these filters only apply for the current viewer and session, but filters may be saved as part of a grid view if desired.
In many cases, the filter popup will open on the Choose Values tab by default. Here, you can directly select one or more individual values using checkboxes. Click on a label to select only a single value, add or remove additional values by clicking on checkboxes.This option is not the default in a few circumstances:
When there are more than 250 unique values in the column, the default option will be Choose Filters. Use filtering expressions to filter. Depending on the datatype of the column, you may still be able to select the Choose Values tab, on which you will see a truncated list of the first 250 values.
Specify a filtering expression (such as "Is Greater Than"), and value (such as "57") and click OK.
You may add a second filter if desired - the second filter is applied as an AND with the first. Both conditions must be true for a row to be included in the filtered results.Once you have filtered on a column, the filter icon appears in the column header. Current filters are listed above the grid, and can be removed by simply clicking the X in the filter panel.When there are multiple active filters, you can remove them individually or use the link to Clear All that will be shown.
Notes:
Leading spaces on strings are not stripped. For example, consider a list filter like Between (inclusive) which takes two comma-separated terms. If you enter range values as "first, second", rows with the value "second" (without the leading space) will be excluded. Enter "first,second" to include such rows.
LabKey filtering is case-sensitive.
Filter Value Variables
Using filter value variables can help you use context-sensitive information in a filter. For example, use the variable "~me~" (including the tildes) on columns showing user names from the core.Users table to filter based on the current logged in user.Additional filter value variables are available that will not work in the grid header menu, but will work in the grid view customizer or in the URL. For example, relative dates can be specified using filter values like "-7d" (7 days ago) or "5d" (5 days from now) in a saved named grid view. Learn more here: Saved Filters and Sorts
Persistent Filters
Some filters on some types of data are persistent (or "sticky") and will remain applied on subsequent views of the same data. For example, some types of assays have persistent filters for convenience; these are listed in the active filter bar above the grid.
Use Faceted Filtering
When applying multiple filters to a data grid, the options shown as available in the filter popup will respect prior filters. For example, if you first filter our sample demographics dataset by "Country" and select only "Uganda", then if you open a second filter on "Primary Language" you will see only "French" and "English" as options - our sample data includes no patients from Uganda who speak German or Spanish. The purpose is to simplify the process of filtering by presenting only valid filter choices. This also helps you avoid unintentionally empty results.
Understand Filter URLs
Filtering specifications can be included on the page URL. A few examples follow.This URL filters the example "PhysicalExam" dataset to show only rows where weight is greater than 80kg. The column name, the filter operator, and the criterion value are all specified as URL parameters. The dataset is specified by ID, "5003" in this case:
Multiple filters on different columns can be combined, and filters also support selecting multiple values. In this example, we show all rows for two participants with a specific data entry date:
Within a study dataset, you may also filter a data grid by participant group. Click the (Filter) icon above the grid to open the filter panel. Select checkboxes in this panel to further filter your data. Note that filters are cumulatively applied and listed in the active filter bar above the data grid.
Filtering expressions available for columns or when searching for subjects of interest will vary by datatype of the column, and not all expressions are relevant or available in all contexts.
In the following tables, the "Arguments" column indicates how many data values, if any, should be provided for comparison with the data being filtered.
Expression
Arguments
Example Usage
Description
Has Any Value
Returns all values, including null
Is Blank
Returns blank values
Is Not Blank
Returns non-blank values
Equals
1
Returns values matching the value provided
Does Not Equal
1
Returns non-matching values
Is Greater Than
1
Returns values greater than the provided value
Is Less Than
1
Returns values less than the provided value
Is Greater Than or Equal To
1
Returns values greater than or equal to the provided value
Is Less Than or Equal To
1
Returns values less than or equal to the provided value
Contains
1
Returns values containing a provided value
Does Not Contain
1
Returns values not containing the provided value
Starts With
1
Returns values which start with the provided value
Does Not Start With
1
Returns values which do not start with the provided value
Between, Inclusive
2, comma separated
-4,4
Returns values between or matching two comma separated values provided
Not Between, Exclusive
2, comma separated
-4,4
Returns values which are not between and do not match two comma separated values provided
Equals One Of
1 or more values, either semicolon or new line separated
a;b;c a b c
Returns values matching any one of the values provided
Does Not Equal Any Of
1 or more values, either semicolon or new line separated
a;b;c a b c
Returns values not matching a provided list
Contains One Of
1 or more values, either semicolon or new line separated
a;b;c a b c
Returns values which contain any one of a provided list
Does Not Contain Any Of
1 or more values, either semicolon or new line separated
a;b;c a b c
Returns values which do not contain any of a provided list
Summary statistics for a column of data can be displayed directly on the grid, giving at-a-glance information like count, min, max, etc. of the values in that column.
Click a column header, then select Summary Statistics.
The popup will list all available statistics for the given column, including their values for the selected column.
Check the box for all statistics you would like to display.
Click Apply. The statistics will be shown at the bottom of the column.
Display Multiple Statistics
Multiple summary statistics can be shown at one time for a column and each column can have it's own set. Here is a compound set of statistics on another dataset:
Statistics Available
The list of statistics available vary based on the edition of LabKey Server you are running, and on the column datatype. Not all functions are available for all column types, only meaningful aggregates are offered. For instance, boolean columns show only the count fields; date columns do not include sums or means. Calculations ignore blank values, but note that values of 0 or "unknown" are not blank values.All calculations use the current grid view and any filters you have applied. Remember that the grid view may be shown across several pages. Column summary statistics are for the dataset as a whole, not just the current page being viewed. The number of digits displayed is governed by the number format set for the container, which defaults to rounding to the thousandths place.Summary statistics available in the Community edition include:
Count (non-blank): The number of values in the column that are not blank, i.e. the total number of rows for which there is data available.
Sum: The sum of the values in the column.
Mean: The mean, or average, value of the column.
Minimum: The lowest value.
Maximum: The highest value.
Additional summary statistics available in Premium editions of LabKey Server include:
Count (blank): The number of blank values.
Count (distinct): The number of distinct values.
Median: Orders the values in the column, then finds the midpoint. When there are an even number of values, the two values at the midpoint are averaged to obtain a single value.
Median Absolute Deviation (MAD): The median of the set of absolute deviations of each value from the median.
Standard Deviation (of mean): For each value, take the difference between the value and the mean, then square it. Average the squared deviations to find the variance. The standard deviation is the square root of the variance.
Standard Error (of mean): The standard deviation divided by the square of the number of values.
Quartiles:
Lower (Q1) is the midpoint between the minimum value and the median value.
Upper (Q3) is the midpoint between the median value and the maximum value. Both Q1 and Q3 are shown.
Interquartile Range: The number of values between Q3 and Q1 in the ordered list. Q3-Q1.
Save Summary Statistics with Grid View
Once you have added summary statistics to a grid view, you can use (Grid Views) > Customize Grid to save the grid with these statistics displayed. For example, this grid in our example study shows a set of statistics on several columns (scroll to the bottom):
For example, this grid in our example study includes both column visualizations and summary statistics:
A grid view is a way to see tabular data in the user interface. Lists, datasets, assay results, and queries can all be represented in a grid view. The default set of columns displayed are not always what you need. This topic explains how to create custom grid views in the user interface, showing selected columns in the order you wish. You can edit the default grid view, and also save as many named grid views as needed.Editors, administrators, and users granted "Shared View Editor" access can create and share customized grid views with other users.
On the Columns tab of the grid view customizer, control the fields included in your grid and how they are displayed in columns:
Available Fields: Lists the fields available in the given data structure.
Selected Fields: Shows the list of fields currently selected for display in the grid.
Delete: Deletes the current grid view. You cannot delete the default grid view.
Revert: Returns the grid to its state before you customized it.
View Grid: Click to preview your changes.
Save: Click to save your changes as the default view or as a new named grid.
Actions you can perform here are detailed below.
Add Columns
To add a column to the grid, check the box for it in the Available Fields pane.
The field will be added to the Selected Fields pane.
This is shown for the "Hemoglobin" field in the above image.
Notice the checkbox for Show hidden fields. Hidden fields might contain metadata about the grid displayed or interconnections with other data. Learn more about common hidden fields for lists and datasets in this topic:
When the underlying table includes elements that reference other tables, they will be represented as an expandable node in the Available Fields panel. For example:
Fields defined to lookup values in other tables, such as the built in "Created By" shown above. In that case to show more information from the Users table.
In a study, there is built in alignment around participant and visit information; see the "Participant Visit" node.
In a study, all the other datasets are automatically joined through a special field named "Datasets". Expand it to see all other datasets that can be joined to this one. Combining datasets in this way is the equivalent of a SQL SELECT query with one or more inner joins.
Click and buttons to expand/collapse nodes revealing columns from other datasets and lookups.
Greyed out items cannot be displayed themselves, but can be expanded to find fields to display.
Add All Columns for a Node
If you want to add all the columns in a given node, hold down shift when you click the checkbox for the top of the node. This will select (or unselect) all the columns in that node.Note that this will only add one "layer" of the columns under a node. It also does not apply to any grayed out "special" nodes like the "Datasets" node in a study.
Reorder Columns
To reorder the columns, drag and drop the fields in the Selected Fields pane. Columns will appear left to right as they are listed top to bottom. Changing the display order for a grid view does not change the underlying data table.
Change Column Display Name
Hover over any field in the Selected Fields panel to see a popup with more information about the key and datatype of that field, as well as a description if one has been added.To change the column display title, click the (Edit title) icon. Enter the new title and click OK. Note that this does not change the underlying field name, only how it is displayed in the grid.
Remove Columns
To remove a column, hover over the field in the Selected Fields pane, and click (Remove column) as shown in the image above.Removing a column from a grid view does not remove the field from the dataset or delete any data.
You can also remove a column directly from the grid (without opening the view customizer) by clicking the column header and selecting Remove Column. After doing so, you will see the "grid view has been modified" banner and be able to revert, edit, or save this change.
Save Grid Views
When you are satisfied with your grid, click Save. You can save your version as the default grid view, or as a new named grid view.
Click Save.
In the popup, select:
Grid View Name:
"Default grid view for this page": Save as the grid view named "Default".
"Named": Select this option and enter a title for your new grid view. This name cannot be "Default" and if you use the same name as an existing grid view, your new version will overwrite the previous grid by that name.
Shared: By default a customized grid is private to you. If you have the "Editor" role or "Shared View Editor" role (or higher) in the current folder, you can make a grid available to all users by checking the box Make this grid view available to all users.
Inherit: If present, check the box to make this grid view available in child folders.
In this example, we named the grid "My Custom Grid View" and it was added to the (Grid Views) pulldown menu.
Saved grid views appear on the (Grid Views) pulldown menu. On this menu, the "Default" grid is always first, then any grids saved for the current user (shown with a icon), then all the shared grid views. If a customized version of the "Default" grid view has been saved but not shared by the current user, it will also show a lock icon, but can still be edited (or shared) by that user.When the list of saved grid views is long, a Filter box is added. Type to narrow the list making it easier to find the grid view you want.In a study, named grid views will be shown in the Data Views webpart when "Queries" are included. Learn more in this topic: databrowser
Save Filtered and Sorted Grids
You can further refine your saved grid views by including saved filters and sorts with the column configuration you select. You can define filters and sorts directly in the view customizer, or get started from the user interface using column menu filters and sorts.Learn about saving filters and sorts with custom grid views in this topic:
Note: When saving or modifying grid views of assay data, note that run filters may have been saved with your grid view. For example, if you customize the default grid view while viewing a single run of data, then import a new run, you may not see the new data because of the previous filtering to the other single run. To resolve this issue:
Select (Grid Views) > Customize Grid.
Click the Filters tab.
Clear the Row ID filter by clicking the 'X' on it's line.
Click Save, confirm Default grid view for this page is selected.
Click Save again.
Include Column Visualizations and Summary Statistics
When you save a default or named grid, any Column Visualizations or Summary Statistics in the current view of the grid will be saved as well. This lets you include quick graphical and statistical information when a user first views the data.For example, this grid in our example study includes both column visualizations and summary statistics:
Every data structure has a grid view named "Default" but it does not have to be the default shown to users who view the structure.
To set the default view to another named grid view, select (Grid Views) > Set Default.
The current default view will be shown in bold.
Click Select for the grid you prefer from the list available. The newly selected on will now be bold.
Click Done.
Revert to the Original Default Grid View
To revert any customizations to the default grid view, open it using (Grid Views) > Default.
Select (Grid Views) > Customize Grid.
Click the Revert button.
Views Web Part
To create a web part listing all the customized views in your folder, an administrator can create an additional web part:
Enter > Page Admin Mode
In the lower left, select Views from the Select Web Part menu.
Click Add.
The web part will show saved grid views, reports, and charts sorted by categories you assign. Here we see the new grid view we just created.
Inheritance: Making Custom Grids Available in Child Folders
In some cases, listed in the table below, custom grids can be "inherited" or made available in child folders. This generally applies to cases like queries that can be run in different containers, not to data structures that are scoped to a single folder. Check the box to Make this grid view available in child folders.The following table types support grid view inheritance:
In a study, why can't I customize my grid to show a particular field from another dataset?Background: To customize your grid view of a dataset by adding columns from another dataset, it must be possible to join the two datasets. The columns used for a dataset's key influence how this dataset can be joined to other tables. Certain datasets have more than one key column (in other words, a "compound key"). In a study, there are three types of datasets, distinguished by how you set Data Row Uniquenesswhen you create the dataset:
Demographic datasets use only the Participants column as a key. This means that only one row can be associated with each participant in such a dataset.
Clinical (or standard) datasets use Participant/Visit (or Timepoint) pairs as a compound key. This means that there can be many rows per participant, but only one per participant at each visit or date.
Datasets with an additional key field including assay or sample data linked into the study. In these compound key datasets, each participant can have multiple rows associated with each individual visit; these are uniquely differentiated by another column key, such as a rowID.
Consequences: When customizing the grid for a table, you cannot join in columns from a table with more key columns. For example, if you are looking at a demographics dataset in a study, you cannot join to a clinical dataset because the clinical dataset can have multiple rows per participant, i.e. has more columns in its key. There isn't a unique mapping from a participant in the 'originating' dataset to a specific row of data in the dataset with rows for 'participant/visit' pairs. However, from a clinical dataset, you can join either to demographic or other clinical datasets.Guidance: To create a grid view combining columns from datasets of different 'key-levels', start with the dataset with more columns in the key. Then select a column from the table with fewer columns in the key. There can be a unique mapping from the compound key to the simpler one - some columns will have repeated values for several rows, but rows will be unique.
Show CreatedBy and/or ModifiedBy Fields to Users
In LabKey data structures, there are built in lookups to store information about the users who create and modify each row. You can add this data to a customized grid by selecting/expanding the CreatedBy and/or ModifiedBy fields and checking the desired information.However, the default lookup for such fields is to the core.SiteUsers table, which is accessible only to site administrators; others see only the row for their own account. If you wish to be able to display information about other users to a non-admin user, you need to customize the lookup to point to the core.Users table instead. That table has the same columns but holds only rows for the users in the current project, restricting how much information is shared about site-wide usage to non-admins. From the schema descriptions:
core.SiteUsers: Contains all users who have accounts on the server regardless of whether they are members of the current project or not. The data in this table are available only to site administrators. All other users see only the row for their own account.
core.Users: Contains all users who are members of the current project. The data in this table are available only to users who are signed-in (not guests). Guests see no rows. Signed-in users see the columns UserId, EntityId, and DisplayName. Users granted the 'See User and Group Details' role see all standard and custom columns.
To adjust this lookup so that non-admins can see CreatedBy details:
Open > Go To Module > Query.
If you don't have access to this menu option, you would need higher permissions to make this change.
Now anyone with the "Reader" role (or higher) will be able to see the display name of the user who created and/or modified a row when it's included in a custom grid view.
How do I recover from a broken view?
It is possible to get into a state where you have a "broken" view, and the interface can't return you to the editor to correct the problem. You may see a message like:
View has errors org.postgresql.util.PSQLException: … details of error
Suggestions:
It may work to log out and back in again, particularly if the broken view has not been saved.
Depending on your permissions, you may be able to access the view manager by URL to delete the broken view. The Folder Administrator role (or higher) is required. Navigate to the container and edit the URL to replace "/project-begin.view?" with:
/query-manageViews.view?
This will open a troubleshooting page where admins can delete or edit customized views. On this page you will see grid views that have been edited and saved, including a line with no "View Name" if the Default grid has been edited and saved. If you delete this row, your grid will 'revert' any edits and restore the original Default grid.
If you are still having issues with the Default view on a grid, try accessing a URL like the following, replacing <my_schema> and <my_table> with the appropriate values:
If you find grid menus to be unresponsive in a LabKey Study dataset, i.e. you can see the menus drop down but clicking options do not have any effect, double check that there are no apostrophes (single quote marks) in the definition of any cohorts defined in that study.Learn more here.
When you are looking at a data grid, you can sort and filter the data as you wish, but those sorts and filters only persist for your current session on that page. Using the .lastFilter parameter on the URL can preserve the last filter, but otherwise these sorts and filters are temporary.To create a persistent filter or sort, you can save it as part of a custom grid view. Users with the "Editor" or "Shared View Editor" role (or higher) can share saved grids with other users, including the saved filters and sorts.
Check a box in the Available Fields panel to add a sort on that field.
In the Selected Sorts panel, specify whether the sort order should be ascending or descending for each sort applied.
Click Save.
You may save as a new named grid view or as the default.
If you have sufficient permissions, you will also have the option to make it available to all users.
You can also create a saved sort by first sorting your grid directly using the column headers, then opening the grid customizer panel to convert the local sort to a saved one.
In the grid view with the saved sort applied above, sort on a second column, in this example we chose 'Lymphs'.
Open (Grid Views) > Customize Grid.
Click the Sort tab. Note that it shows (2), meaning two sorts are now defined. Until you save the grid view with this additional sort included, it will remain temporary, as sorts usually are.
Drag and drop if you want to change the order in which sorts are applied.
Remember to Save your grid to save the additional sort.
Define a Saved Filter
The process for defining saved filters is very similar. You can filter locally first or directly define saved filters.
An important advantage of using the saved filters interface is that when filtering locally, you are limited to two filters on a given column. Saved filters may include any number of separate filtering expressions for a given column, which are all ANDed together.
Another advantage is that there are some additional filtering expressions available here that are not available in the column header filter dialog.
For example, you can filter by relative dates using -#d syntax (# days ago) or #d (# days from now) using the grid customizer but not using the column header.
Select (Grid Views) > Customize Grid.
Click the Filter tab.
In the left panel, check boxes for the column(s) on which you want to filter.
Drag and drop filters in the right panel to change the filtering order.
In the right panel, specify filtering expressions for each selected column. Use pulldowns and value entry fields to set expressions, add more using the (Add) buttons.
Use the buttons to delete individual filtering expressions.
Save the grid, selecting whether to make it available to other users.
Apply Grid Filter
When viewing a data grid, you can enable and disable all saved filters and sorts using the Apply Grid Filter checkbox in the (Grid Views) menu. All defined filters and sorts are applied at once using this checkbox - you cannot pick and choose which to apply. If this menu option is not available, no saved filters or sorts have been defined.Note that clearing the saved filters and sorts by unchecking this box does not change how they are saved, it only clears them for the current user and session.
Interactions Among Filters and Sorts
Users can still perform their own sorting and filtering as usual when looking at a grid that already has a saved sort or filter applied.
Sorting: Sorting a grid view while you are looking at it overrides any saved sort order. In other words, the saved sort can control how the data is first presented to the user, but the user can re-sort any way they wish.
Filtering: Filtering a grid view which has one or more saved filters results in combining the sets of filters with an AND. That is, new local filters happen on the already-filtered data. This can result in unexpected results for the user in cases where the saved filter(s) exclude data that they are expecting to see. Note that these saved filters are not listed in the filters bar above the data grid, but they can all be disabled by unchecking the (Grid Views) > Apply View Filter checkbox.
When you work with a grid of data, such as a list or dataset, you often need to select one or more rows. For example, you may wish to visualize a subset of data or select particular rows from an assay to link into a study. Large data grids are often viewed as multiple pages, adding selection options.Topics on this page:
To select any single row, click the checkbox at the left side of the row.
To unselect the row, uncheck the same checkbox.
The box at the top of the checkbox column allows you to select or unselect all rows on the current page at once.
Select Rows on Multiple Pages
Using the and buttons in the top right of the grid, you can page forward and back in your data and select as many rows as you like, singly or by page, using the same checkbox selection methods as on a single page. The selection message will update showing the total tally of selected rows.To change the number of rows per page, select the row count message ("1 - 100 of 677" in the above screencap) to open a menu. Select Paging and make another selection for rows per page. See Page Through Data for more.
Selection Buttons
Selecting the box at the top of the checkbox column also adds a bar above your grid which indicates the number of rows selected on the current page and additional selection buttons.
Select All Selectable Rows: Select all rows in the dataset, regardless of pagination.
Select None: Unselect all currently selected rows.
Show All: Show all rows as one "page" to simplify sorting and selection.
Show Selected: Show only the rows that are selected in a single page grid.
Show Unselected: Show only the rows that are not selected in a single page grid.
Using the Show Selected option can be helpful in keeping track of selections in large datasets, but is also needed for some actions which may only apply to rows on the current page which are selected.
LabKey provides a variety of methods for exporting the rows of a data grid. You can export into formats that can be consumed by external applications (e.g., Excel) or into scripts that can regenerate the data grid. You can also choose whether to export the entire set of data or only selected rows.
Your choice of export format determines whether you get a static snapshot of the data, or a dynamic reflection that updates as the data changes. The Excel and TSV formats supply static snapshots, while scripts allow you to display dynamically updated data.
Premium Feature AvailableSubscribers to the Enterprise Edition of LabKey Server can export data with an e-Signature as described in this topic:
Click the (Export) button above any grid view to open the export panel.
Use tabs to choose between Excel, Text and Script exports, each of which carries a number of appropriate options for that type.
After selecting your options, decribed below, and clicking the Export button, you will briefly see visual feedback that the export is in progress:
Export Column Headers
Both Excel and Text exports allow you to choose whether Column Headers are exported with the data, and if so, what format is used. Options:
None: Export the data table with no column headers.
Caption: (Default) Include a column header row using the currently displayed column captions as headers.
Field Key: Use the column name with FieldKey encoding. While less display friendly, these keys are unambiguous and canonical and will ensure clean export and import of data into the same dataset.
Export Selected Rows
If you select one or more rows using the checkboxes on the left, you will activate the Export Selected Rows checkbox in either Excel or Text export mode. When selected, your exported Excel file will only include the selected rows. Uncheck the box to export all rows. For additional information about selecting rows, see Select Rows.
Filter Data Before Export
Another way to export a subset of data records is to filter the grid view before you export it.
Filter Data. Clicking a column header in a grid will open a dialog box that lets you filter and exclude certain types of data.
Create or select a Custom Grid View. Custom Grids let you store a selected subset as a named grid view.
View Data From One Visit. You can use the Study Navigator to view the grid of data records for a particular visit for a particular dataset. From the Study Navigator, click on the number at the intersection of the desired visit column and dataset row.
Export to Excel
When you export your data grid to Excel, you can use features within that software to access, sort and present the data as required. If your data grid includes inline images they will be exported in the cell in which they appear in the grid.
Export to Text
Select Text tab to export the data grid in a text format. Select tab, comma, colon, or semicolon from the Separator pulldown and single or double from the Quote pulldown. The extension of your exported text file will correspond to the separator you have selected, i.e. "tsv" for tab separators.LabKey Server uses the UTF-8 character encoding when exporting text files.
Export to Script
You can export the current grid to script code that can be used to access the data from any of the supported client libraries. See Export Data Grid as a Script.The option to generate a Stable URL for the grid is also included on the (Export) > Script tab.
Related Topics
Export Assay Data - Additional export options are available for assay data.
The default dataset grid displays data for all participants. To view data for an individual participant, click on the participantID in the first column of the grid. The Participant Details View can also be customized to show graphical or other custom views of the data for a single study subject.
The participant details view lists all of the datasets that contain data for the current participant, as shown in the image below.
Previous/Next Participant: Page through participants. Note that this option is only provided when you navigate from a dataset listing other participants. Viewing a single participant from the Participants tab does not include these options.
button: Expand dataset details.
button: Collapse dataset details
Add Charts
Expand dataset details by clicking the or name of the dataset of interest. Click Add Chart to add a visualization to the participant view details. The dropdown will show the charts defined for this dataset.After you select a chart from the dropdown, click the Submit button that will appear.Once you create a chart for one participant in a participant view, the same chart is displayed for every participant, with that participant's data.You can add multiple charts per dataset, or different charts for each dataset. To define new charts to use in participant views, use the plot editor.
Notes:1. Charts are displayed at a standard size in the default participant details view; custom height and width, if specified in the chart definition, are overridden.2. Time charts displaying data by participant group can be included in a participant details view, however the data is filtered to show only data for the individual participant. Disregard the legend showing group names for these trend lines.
Customize Participant Details View
You can alter the HTML used to create the default participant details page and save alternative ways to display the data using the Customize View link.
You can make small styling or other adjustments to the "default" script. An example is below
You can leverage the LabKey APIs to tailor your custom page.
You can also add the participant.html file via a module. Learn more about file based modules in this webinar: Tech Talk: Custom File-Based Modules
Click Save to refresh and see the preview below your script. Click Save and Finish when finished.
Example 1: Align Column of Values
In the standard participant view, the column of values for expanded demographics datasets will be centered over the grid of visits for the clinical datasets. In some cases, this will cause the demographic field values to be outside the visible page. To change where they appear, you can add the following to the standard participant details view (under the closing </script> tag):
Use a wider "width" value if you have very long field names in the first column.
Example 2: Tabbed Participant View (Premium Resource)
Another way a developer can customize the participant view is by using HTML with JavaScript to present a tabbed view of participant data. Click the tabs to see the data for a single participant in each category of datasets; step through to see data for other participants.
Premium Resource AvailableSubscribers to premium editions of LabKey Server can use the example code in this topic to create views like the tabbed example above:
If you see field names but no values in the default participant view, particularly for demographic datasets, check to be sure that your field names do not include special characters. If you want to present fields to users with special characters in the names, you can use the "Label" of the field to do so.
For certain LabKey queries, including assay designs, issue trackers, and survey designs, but not including study datasets, the scope of the query can be set with a folder filter to include:
all data on the site
all data for a project
only data located in particular folders
in some cases, combinations of data including /Shared project data
For example, the query itself can be defined at the project level. Data it queries may be located within individual subfolders. Scope is controlled using the "Filter by Folder" option on the (Grid Views) menu in the web part.This allows you to organize your data in folders that are convenient to you at the time of data collection (e.g., folders for individual labs or lab technicians). Then you can perform analyses independently of the folder-based organization of your data. You can analyze data across all folders, or just a branch of your folder tree.You can set the scope through either the (Grid Views) menu or through the client API. In all cases, LabKey security settings remain in force, so users only see data in folders they are authorized to see.
Some types of grid may have different selections available. For example, Sample Types, Data Classes, and Lists offer the following option to support cross-folder lookups:
Current folder, project, and Shared project
Folder Filters in the JavaScript API
The LabKey API provides developers with even finer grained control over the scope.The containerFilter config property is available on many methods on such as LABKEY.Query.selectRows, provides fine-grained control over which folders are accessed through the query.For example, executeSQL allows you to use the containerFilter parameter to run custom queries across data from multiple folders at once. A query might (for example) show the count of NAb runs available in each lab’s subfolder if folders are organized by lab.Possible values for the containerFilter are:
allFolders
current
currentAndFirstChildren
currentAndParents
currentAndSubfolders
currentPlusProject
currentPlusProjectAndShared
Folder Filters with the JDBC Driver
The LabKey JDBC Driver also supports the containerFilter parameter for scoping queries by folder. Learn more in this topic:
The right visualization can communicate information about your data efficiently. You can create different types of report and chart to view, analyze and display data using a range of tools.
Basic visualization types available to non-administrative users from the (Charts) > Create Chart menu and directly from data grid column headers are described in this topic:
LabKey Server can serve as the data store for external reporting and visualization tools such as RStudio, Tableau, Access, Excel, etc. See the following topics for details.
Premium Feature — Available in the Professional and Enterprise Editions of LabKey Server. Learn more or contact LabKey.
This topic describes how to import reports from Jupyter Notebooks and use them with live data in LabKey Server. By configuring a Jupyter scripting engine, you can add a Jupyter Reports option on the menu of data grids.
In previous versions, this feature was enabled by configuring a docker image and a "Docker Report Engine". You can still use the same docker image configuration and use the resulting "localhost" URL for the "Jupyter Report Engine".
Once integrated, users can access the Jupyter report builder to import .ipynb files exported from Jupyter Notebooks. These reports can then be displayed and shared in LabKey.
The Jupyter scripting configuration requires the URL of the service endpoint. Depending on where your Jupyter service is running, it could be something like "http://localhost:3031/" or "http://service-prefix-jupyter.srvc.int.jupyter:3031"
If you are connecting to a docker image, find the Remote URL by going to the Settings tab of the Admin Console. Under Premium Features, click Docker Host Settings. The "Docker Host" line shows your Remote URL.
The URL "http://noop.test:3031" can be used to configure a fake 'do nothing' service. This is only useful for testing.
Add Jupyter Report Engine on LabKey Server
Select > Site > Admin Console.
Under Configuration, click Views and Scripting.
Select Add > New Jupyter Report Engine. In the popup, you'll see the default options and can adjust them if needed.
Language: Python
Language Version: optional
File extension: ipynb (don't include the . before the extension)
Click Submit to save.You can now see the Jupyter Reports menu option in the Data Views web part and on the grid menu where you created it.
Obtain Report .ipynb from Jupyter Notebooks
Within Jupyter Notebooks, you can now open and author your report. You can use the LabKey Python API to access data and craft the report you want. One way to get started is to export the desired data in a python script form that you can use to obtain that data via API.Use this script as the basis for your Jupyter Notebook report.When ready, you'll "export" the report by simply saving it as an .ipynb file. To save to a location where you can easily find it for importing to LabKey, choose 'Save As' and specify the desired location.
Create Jupyter Report on LabKey Server
From any data grid, or from the Data Views web part, you can add a new Jupyter Report. You'll see a popup asking if you want to:
Click Import From File in the popup and browse to select the .ipynb file to open. You'll see your report text on the Source tab of the Jupyter Report Builder.Click Save to save this report on LabKey. Enter a report name and click OK. Now your report will be accessible from the menu.
Start with Blank Report
The blank report option offers a basic wrapper for building your report.
Jupyter Report Builder
The Report Builder offers tabs for:
Report: See the report.
Data: View the data on which the report is based.
Source: Script source, including Save and Cancel for the report. Options are also available here:
Make this report available to all users
Show source tab to all users
Make this report available in child folders
Help: Details about the report configuration options available.
Help Tab - Report Config Properties
When a Jupyter Report is executed, a config file is generated and populated with properties that may be useful to report authors in their script code. The file is written in JSON to : report_config.json.
A helper utility : ReportConfig.py is included in the nbconfig image. The class contains functions that will parse the generated file and return configured properties to your script. An example of the code you could use in your script:
from ReportConfig import get_report_api_wrapper, get_report_data, get_report_parameters print(get_report_data()) print(get_report_parameters())
This is an example of a configuration file and the properties that are included.
The Jupyter report in LabKey can also be exported as an .ipynb file for use elsewhere. Open the report, choose the Source tab, and under Jupyter Report Options, click Export.
Displaying a report or chart alongside other content helps you highlight visualizations of important results. There are a number of ways to do this, including:
Display a saved report or chart using the Report web part. This topic covers this option.
Click Add Web Part in the lower left, select Report, and click Add.
On the Customize Report page, enter the following parameters:
Web Part Title: This is the title that will be displayed in the web part.
Report or Chart: Select the report or chart to display.
Show Tabs: Some reports may be rendered with multiple tabs showing.
Visible Report Sections: Some reports contain multiple sections, such as: images, text, console output. If a list is offered, you can select which section(s) to display by selecting them. If you are displaying an R Report, the sections are identified by the section names from the source script.
Click Submit.
In this example, the new web part will look like this:
Change Report Web Part Settings
You can reopen the Customize Report page later to change the name or how it appears.
Select Customize from the (triangle) menu.
Click Submit.
Options available (not applicable to all reports):
Show Tabs: Some reports may be rendered with multiple tabs showing. Select this option to only show the primary view.
Visible Report Sections: Some reports contain multiple sections such as: images, text, console output. For these you can select which section(s) to display by selecting them from the list.
The Data Views web part displays a catalog of available reports and custom named grid views. This provides a convenient dashboard for selecting among the available ways to view data in a given folder or project. In a Study the Data Views web part also includes datasets. It is shown on the Clinical and Assay Data tab by default, but can be added to other tabs or pages as needed.
By default, the Data Views web part lists all the custom grid views, reports, etc. you have permission to read. If you would like to view only the subset of items you created yourself, click the Mine checkbox in the upper right.
Add and Manage Content
Users with sufficient permission can add new content and make some changes using the menu in the corner of the web part. Administrators have additional options, detailed below.
Add Report: Click for a submenu of report types to add.
Add Chart: Use the plot editor to add new visualizations.
Manage Views: Manage reports and custom grids, including the option to delete multiple items at once. Your permissions will determine your options on this page.
Manage Notifications: Subscribe to receive notifications of report and dataset changes.
Toggle Edit Mode
Users with Editor (or higher) permissions can edit metadata about items in the databrowser.
Click the pencil icon in the web part border to toggle edit mode. Individual pencil icons show which items have metadata you can edit here.
When active, click the pencil icon for the desired item.
Edit Properties, such as status, author, visibility to others, etc.
If you want to move the item to a different section of the web part, select a different Category.
If there are associated thumbnails and mini-icons, you can customize them from the Images tab. See Manage Thumbnail Images for more information.
Click Save.
Notice that there are three dates associated with reports: the creation date, the date the report itself was last modified, and the date the content of the report was last modified.
Data Views Web Part Options
An administrator adds the Data Views web part to a page:
Permissions: Control what permissions a user must have to see this web part.
Move Up/Down: Change the sequence of web parts on the page. (Only available when an admin is in page admin mode).
Remove From Page: No longer show this web part - note that the underlying data is not affected by removing the web part. (Only available when an admin is in page admin mode).
Hide Frame - remove the header and border of the web part. (Only available when an admin is in page admin mode).
Customize the Data Views Browser
Administrators can also customize display parameters within the web part.
Select Customize from the triangle pulldown menu to open the customize panel:You can change the following:
Name: the heading of the web part (the default is "Data Views").
Display Height: adjust the size of the web part. Options:
Default (dynamic): by default, the data views browser is dynamically sized to fit the number of items displayed, up to a maximum of 700px.
Custom: enter the desired web part height. Must be between 200 and 3000 pixels.
Sort: select an option:
By Display Order: (Default) The order items are returned from the database.
Alphabetical: Alphabetize items within categories; categories are explicitly ordered.
View Types: Check or uncheck boxes to control which items will be displayed. Details are below.
Visible Columns: Check and uncheck boxes to control which columns appear in the web part.
Manage Categories: Click to define and use categories and subcategories for grouping.
To close the Customize box, select Save or Cancel.
Show/Hide View Types
In the Customize panel, you can check or uncheck the View Types to control what is displayed.
Datasets: All study datasets, unless the Visibility property is set to Hidden.
Queries: Customized named grid views on any dataset.
Note that SQL queries defined in the UI or included in modules are not included when this option is checked.
A named grid view created on a Hidden dataset will still show up in the data views browser when "queries" are shown.
Named grid views are categorized with the dataset they were created from.
A query snapshot captures a data query at a moment in time. The data in the snapshot will remain fixed even if the original source dataset is updated until/unless it is refreshed manually or automatically.
You can refresh the resulting query snapshot manually or set up a refresh schedule. If you choose automatic refresh, the system will listen for changes to the original data, and will update the snapshot within the interval of time you select.Snapshotting data in this fashion is only available for:
Queries exposed by linked schemas are not available for snapshotting.
Create a Query Snapshot
Go to the query, grid, or dataset you wish to snapshot.
Select (Charts/Reports) > Create Query Snapshot.
Name the snapshot (the default name appends the word "Snapshot" to the name of the grid you are viewing).
Specify Manual or Automatic Refresh. For automatic refresh, select the frequency from the dropdown. When data changes, the snapshot will be updated within the selected interval of time. Options:
30 seconds
1 minute
5 minutes
10 minutes
30 minutes
1 hour
2 hours
If you want to edit the properties or fields in the snapshot, click Edit Dataset Definition to use the Dataset Designer before creating your snapshot.
Be sure to save (or cancel) your changes in the dataset designer to return to the snapshot creation UI.
Click Create Snapshot.
View a Query Snapshot
Once a query snapshot has been created it is available in the data browser and at > Manage Study > Manage Datasets.
Edit a Query Snapshot
The fields for a query snapshot, as well as the refresh policy and frequency can be edited starting from the grid view.
Select (Grid Views) > Edit Snapshot.
You can see the name and query source here, but cannot edit them.
If you want to edit the fields in the snapshot, (such as to change a column label, etc.) click Edit Dataset Definition to use the Dataset Designer. You cannot change the snapshot name itself. Be sure to save (or cancel) your changes in the dataset designer to return to the snapshot creation UI.
Change the Snapshot Refresh settings as needed.
Click Update Snapshot to manually refresh the data now.
Click Save to save changes to refresh settings.
Click Done to exit the edit interface.
The Edit Snapshot interface also provides more options:
Delete Snapshot
Show History: See the history of this snapshot.
Edit Dataset Definition: Edit fields and their properties for this snapshot. Note that you must either save or cancel your changes in the dataset designer in order to return to the snapshot editor.
Troubleshooting
If a snapshot with the desired name already exists, you may see an error similar to:
If you believe this snapshot does not exist, but are unable to find it in the user interface or schema browser, it may need to be deleted directly from the database. Contact your Account Manager if you need assistance.
Attachment reports enable you to upload and attach stand-alone documents, such as PDF, Word, or Excel files. This gives you a flexible way to interconnect your information.You can create a report or visualization using a statistical or reporting tool outside of LabKey, then upload the report directly from your local machine, or point to a file elsewhere on the server.
Add an Attachment Report
To upload an attachment report, follow these steps:
Create the desired report and save it to your local machine.
Open the (triangle) menu in the Data Views web part.
Select Add Report > Attachment Report.
Provide the name, date, etc. for the report.
Upload the report from your local machine (or point to a document already on the server).
Once the file is uploaded it will be shown in the data browser. If you specify that it is to be shared, other users can view and download it.If the report was saved in the external application with an embedded JPEG thumbnail, LabKey Server can in some cases extract that and use it as a preview in the user interface. See Manage Thumbnail Images for more information.
With a LabKey study, creating a participant report lets you show data for one or more individual participants for selected measures. Measures from different datasets in the study can be combined in a single report. A filter panel lets you dynamically change which
Create a Participant Report
In a study, select > Manage Views.
Choose Add Report > Participant Report.
Click Choose Measures, select one or more measures, then click Select.
Enter a Report Name and optional Report Description.
Select whether this report will be Viewable By all readers or only yourself.
When you first create a report, you will be in "edit mode" and can change your set of chosen measures. Below the selection panel, you will see partial results.
Toggle the edit panel by clicking the (pencil) icon at the top of the report to see more results; you may reopen it at any time to further edit or save the report.
Click the Filter Report (chevron) button to open the filter panel to refine which participants appear in the report.
Select desired filters using the radio buttons and checkboxes. You may hide the filter panel with the , or if you click the X to close it entirely, a Filter Report link will appear on the report menu bar.
Click the Transpose button to flip the columns and rows in the generated tables, so that columns are displayed as rows and vice versa, as shown below.
When you have created the report you want, click Save.
Query reports let you package a query or grid view as a report. This can enable sharing the report with a different audience or presenting the query as of a specific data cut date. A user needs the "Author" role or higher to create a query report. The report also requires that the target query already exists.
Consider naming the report for the query (or grid view) it is "reporting" to make it clearer to the user.
Author: Select from all project users listed in the dropdown.
Status: Choose one of "None, Draft, Final, Locked, Unlocked".
Data Cut Date: Specify a date if desired.
Category: If you are creating a report in a study, you can select an existing category.
Description: Optional text description.
Consider including details like the origin of the report (schema/query/view) so that you can easily search for them later.
Shared: Check the box if you want to share this report with other users.
Schema(Required): The schema containing the desired query. This choice will populate the Query dropdown.
Query(Required): Select a query from those in the selected schema. This will populate the View dropdown.
View: If there are multiple views defined on the selected query, you'll be able to choose one here.
Click Save when finished.
You can customize the thumbnail and mini-icon displayed with your Query Report. Learn more here.
Use and Share Query Reports
Your report is now available for display in a web part or wiki, or sharing with others. Query Reports can be assigned report-specific permissions in a study following the instructions in this topic.In a study, your report will appear in the Data Views web part under the selected category (or as "Uncategorized"). If you want to hide a Query Report from this web part, you can edit the view metadata to set visibility to "Hidden".
Manage Query Reports
Metadata including the name, visibility, data cut date, category and description of a Query Report can all be edited through the manage views interface.To see the details of the schema, query, and view used to create the report, you can view the Report Debug Information page. Navigate to the report itself, you will see a URL like the following, where ### is the id number for your report:
/reports-renderQueryReport.view?reportId=db%3A###
Edit the URL to replace renderQueryReport.view with reportInfo.view, as so:
Reports, charts, datasets, and customized data grids are all ways to view data in a folder and can be displayed in a Data Views web part. Within a study, this panel is displayed on the Clinical and Assay Data tab by default, and can be customized or displayed in other places as needed. This topic describes how to manage these "data views".
Select > Manage Views in any folder. From the Data Views web part, you can also select it from the (triangle) pulldown menu.The Manage Views page displays all the views, queries, and reports available within a folder. This page allows editing of metadata as well as deletion of multiple items in one action.
A row of links are provided for adding, managing, and deleting views and attributes like categories and notifications.
Filter by typing part of the name, category, type, author, etc. in the box above the grid.
By default you will see all queries and reports you can edit. If you want to view only items you created yourself, click the Mine checkbox in the upper right.
Hover over the name of an item on the list to see a few details, including the type, creator, status, and an optional thumbnail.
Click on the name to open the item.
Click a Details link to see more metadata details.
Notice the icons to the right of charts, reports, and named grids. Click to edit the metadata for the item.
When managing views within a study, you can click an active link in the Access column to customize permissions for the given visualization. "Public" in this column refers to the item being readable by users with at least Read access to the container, not to the public at large unless that is separately configured. For details see Configure Permissions for Reports & Views.
View Details
Hover over a row to view the source and type of a visualization, with a customizable thumbnail image.Clicking the Details icon for a report or chart opens the Report Details page with the full list of current metadata. The details icon for a query or named view will open the view itself.
Modification Dates
There are two modification dates associated with each report, allowing you to differentiate between report property and content changes:
Modified: the date the report was last modified.
Name, description, author, category, thumbnail image, etc.
Content Modified: the date the content of the report was modified.
Underlying script, attachment, link, chart settings, etc.
The details of what constitutes content modification are report specific:
Script Reports including JavaScript and R Reports:
Change to the report code (JavaScript, R, etc.)
Flow Reports (Positivity and QC):
Change to any of the filter values
The following report types do not change the ContentModified date after creation: Crosstab View, DataReport, External Report, Query Report, Chart Reports, Chart View.
Edit View Metadata
Click the pencil icon next to any row to edit metadata to provide additional information about when, how, and why the view or report was created. You can also customize how the item is displayed in the data views panel.
View Properties
Click the pencil icon to open a popup window for editing visualization metadata. On the Properties tab:
Modify the Name and Description fields.
Select Author, Status, and Category from pulldown lists of valid values. For more about categories, see Manage Categories.
Choose a Data Cut Date from the calendar.
Check whether to share this report with all users with access to the folder.
Click Save.
The Images tab is where you modify thumbnails and mini-icons used for the report.You could also delete this visualization from the Properties tab by clicking Delete. This action is confirmed before the view is actually deleted.
View Thumbnails and Mini-icons
When a visualization is created, a default thumbnail is auto-generated and a mini-icon based on the report type is associated with it. You can see and update these on the Images tab. Learn more about using and customizing these images in this topic:
To rearrange the display order of reports and charts, click Reorder Reports and Charts. Users without administrator permissions will not see this button or be able to access this feature.Click the heading "Reports and Charts" to toggle searching ascending or decending alphabetically. You can also drag and drop to arrange in any order.When the organization is correct, click Done.
File based reports can be moved within the dialog box, but the ordering will not actually change until you make changes to their XML.
Delete Views and Reports
Select any row by clicking an area that is not a link. You can use Shift and Ctrl to multi-select several rows at once. Then click Delete Selected. You will be prompted to confirm the list of the items that will be deleted.
Manage Study Notifications
Users can subscribe to a daily digest of changes to reports and datasets in a study. Learn more in this topic: Manage Study Notifications
If you want to receive email notifications when the reports and/or datasets in a study are updated, you can subscribe to a daily digest of changes. These notifications are similar to email notifications for messages and file changes at the folder level, but allow finer control of which changes trigger notification.
Select Manage Notifications from the (triangle) menu in the Data Views web part.
You can also select > Manage Views then click Manage Notifications.
Select the desired options:
None. (Default)
All changes: Your daily digest will list changes and additions to all reports and datasets in this study.
By category: Your daily digest will list changes and additions to reports and datasets in the subscribed categories.
By dataset: Your daily digest will list changes and additions to subscribed datasets. Note that reports for those datasets are not included in this option.
Click Save.
You will receive your first daily digest of notifications at the next system maintenance interval, typically overnight. By default, the notification includes the list of updated reports and/or datasets including links to each one.
Subscribe by Category
You can subscribe to notifications of changes to both reports and datasets by grouping them into categories of interest. If you want to allow subscription to notifications for a single report, create a singleton subcategory for it.Select the By Category option, then click checkboxes under Subscribe for the categories and subcategories you want to receive notifications about.
Subscribe by Dataset
To subscribe only to notifications about specific datasets, select the By dataset option and use checkboxes to subscribe to the datasets of interest.Note that reports based on these datasets are not included when you subscribe by dataset.
Notification Triggers
The following table describes which data view types and which changes trigger notifications:
Data Insert/Update/Delete
Design Change
Sharing Status Change
Category Change
Display Order Change
Datasets (including linked Datasets)
Yes
Yes
Yes
Yes
Yes
Query Snapshot
Yes (Notification occurs when the snapshot is refreshed)
Yes
Yes
Yes
Yes
Report (R, JavaScript, etc.)
No
Yes
Yes
Yes
Yes
Chart (Bar, Box Plot, etc.)
No
Yes
Yes
Yes
Yes
Datasets:
Changes to both the design and the underlying data will trigger notifications.
Reports:
Reports must be both visible and shared to trigger notifications.
Only changes to the definition or design of the report, such as changes to the R or JS code, will trigger notifications.
Notifications are generated for all items when their sharing status is changed, their category is changed, or when their display order is changed.
Customize Email Notification
By default, the notification includes the list of updated reports and/or datasets including links to each one. The email notification does not describe the nature of the change, only that some change has occurred.The template for these notifications may be customized at the site-level, as described in Email Template Customization.
In the Data Views web part, reports, visualizations, queries, and datasets can be sorted by categories and subcategories that an administrator defines. Users may also subscribe to notifications by category.Categories can be defined from the Manage Categories page, defined during dataset creation or editing, or during the process of linking data into a study from an assay, or sample type.
Click Manage Categories to open the categories pop-up.
Click New Category to add a category; click the X to delete one, and drag and drop to reorganize sections of reports and grid views.To see subcategories, select a category in the popup. Click New Subcategory to add new ones. Drag and drop to reorder. Click Done in the category popup when finished.
Dataset Designer Interface
During creation or edit of a dataset definition, you can add new categories by typing the new category into the Category box. Click Create option "[ENTERED_NAME]" to create the category and assign the current dataset to it.
Assign Items to Categories
To assign datasets, reports, charts, etc to categories and subcategories, click the (pencil) icon on the manage views page, or in "edit" mode of the data browser. The Category menu will list all available categories and subcategories. Make your choice and click Save.To assign datasets to categories, use the Category field on the dataset properties page. Existing categories will be shown on the dropdown.
Hide a Category
Categories are only shown in the data browser or dataset properties designer when there are items assigned to them. Assigning all items to a different category or marking them unassigned will hide the category. You can also mark each item assigned inside of it as "hidden". See Data Views Browser.
Categorize Linked Assay and Sample Data
When you link assay or sample data into a study, a corresponding Dataset is created (or appended to if it already exists).During the manual link to study process, or when you add auto-linking to the definition of the assay or sample type, you can Specify Linked Dataset Category.
If the study dataset you are linking to already exists and already has a category assignment, this setting will not override the previous categorization. You can manually change the dataset's category directly.
If you are creating a new linked dataset, it will be added to the category you specify.
If the category does not exist, it will be created.
When a visualization is created, a default thumbnail is automatically generated and a mini-icon based on the report or chart type is associated with it. These are displayed in the data views web part. You can customize both to give your users a better visual indication of what the given report or chart contains. For example, rather than have all of your R reports show the default R logo, you could provide different mini-icons for different types of content that will be more meaningful to your users.Attachment Reports offer the additional option to extract the thumbnail image directly from some types of documents, instead of using an auto-generated default.
Enter Edit Mode by clicking the pencil icon in the data views browser or on the manage views page.
Click the pencil icon for any visualization to open the window for editing metadata.
Click the Images tab. The current thumbnail and mini-icon are displayed, along with the option to upload different ones from your local machine.
A thumbnail image will be scaled to 250 pixels high.
A mini-icon will be scaled to 18x18 pixels.
The trash can button will delete the default generated thumbnail image, replacing it with a generic image.
If you have customized the thumbnail, the trash can button, deletes it and returns the default, generated thumbnail image.
Click Save to save any changes you make.
You may need to refresh your browser after updating thumbnails and icons. If you later change and resave the visualization, or export and reimport it with a folder or study, the custom thumbnails and mini-icons will remain associated with it unless you explicitly change them again.
Extract Thumbnails from Documents
An Attachment Report is created by uploading an external document. Some documents can have embedded thumbnails included, and LabKey Server can in some cases extract those thumbnails to associate with the attachment report.The external application, such as Word, Excel, or PowerPoint, must have the "Save Thumbnail" option set to save the thumbnail of the first page as an extractable jpeg image. When the Open Office XML format file (.docx, .pptx, .xlsx) for an attachment report contains such an image, LabKey Server will extract it from the uploaded file and use it as the thumbnail.Images in older binary formats (.doc, .ppt, .xls) and other image formats, such as EMF or WMF, will not be extracted; instead the attachment report will use the default auto-generated thumbnail image.
Measures are values that functions work on, generally numeric values such as instrument readings. Dimensions are qualitative properties and categories, such as cohort, that can be used in filtering or grouping those measures. LabKey Server can be configured such that only specific columns marked as data "measures" or "dimensions" are offered for charting. When this option is disabled, all numeric columns are available for charting and any column can be included in filtering.This topic explains how to mark columns as measures and dimensions, as well as how to enable and disable the restriction.
Dimension: "dimension" means a column of non-numerical categories that can be included in a chart, such as for grouping into box plots or bar charts. Cohort and country are examples of dimensions.
Measure: A column with numerical data. Instrument readings, viral loads, and weight are all examples of measures.
Note: Text columns that include numeric values can also be marked as measures. For instance, a text column that includes a mix of integers and some entries of "<1" to represent values that are below the lower limit of quantitation (LLOQ) could be plotted ignoring the non numeric entries. The server will make a best effort to convert the data to numeric values and display a message about the number of values that cannot be converted.
If your server restricts charting to only measures and dimensions, you have two options: (1) either mark the desired column as a measure/dimension or (2) turn off the restriction.
Mark a Column as a Measure or Dimension
Use the Field Editor, Advanced Settings to mark columns. The method for opening the field editor varies based on the data structure type, detailed in the topic: Field Editor.
Open the field editor.
Expand the field you want to mark.
Click Advanced Settings.
Check the box or boxes desired:
"Make this field available as a measure"
"Make this field available as a dimension"
Click Apply, then Finish to save the changes.
Turn On/Off Restricting Charting to Measures and Dimensions
Note that you must have administrator permissions to change these settings. You can control this option at the site or project level.
To locate this control at the site level:
Select > Site > Admin Console.
Under Configuration, click Look and Feel Settings.
To locate this control at the project level:
Select > Folder > Project Settings.
Confirm that you are viewing the Properties tab (it opens by default).
Scroll down to Restrict charting columns by measure and dimension flags.
You can visualize, analyze and display data using a range of plotting and reporting tools. The right kind of image can illuminate scientific insights in your results far more easily than words and numbers alone. The topics in this section describe how to create different kinds of chart using the common plot editor.
Plot Editor
When viewing a data grid, select (Charts) > Create Chart menu to open the plot editor and create new:
Once created and saved, visualizations will be re-generated by re-running their associated scripts on live data. You can access saved visualizations either through the (Reports or Charts) pulldown menu on the associated data grid, or directly by clicking on the name in the Data Views web part.
Related Topics
Reports and Charts: Additional report and chart types, plus options for managing visualizations.
A bar plot is a visualization comparing measurements of numeric values across categories. The relative size of the bar indicates the relationship between the variable for groups like cohorts in a study.
Select the column of data to use for separating the data into bars and drag it to the X Axis Categories box.
Only the X Axis Categories field is required to create a basic bar chart. By default, the height of the bar shows the count of rows matching each value in the chosen category. In this case the number of participants from each country.
To use a different metric for bar height, select another column and drag it to the box for the Y Axis column. Notice that you can select the aggregate method to use. By default, SUM is selected and the label reads "Sum of [field name]". Here we change to "Mean"; the Y Axis label will update automatically.
Click Apply.
Bar Chart Customizations
To remove various values from your chart, such as if your data includes a large number of "blank" values:
Click View Data.
Click the relevant column header, then select Filter.
Click the checkbox for "Blank" to deselect it.
Click OK in the popup.
Click View Chart to return to the chart which is re-calculated without the data you filtered out.
To make a grouped bar chart, we'll add data from another column.
Click Chart Type to reopen the creation dialog.
Drag a column to the Split Categories By selection box.
Click Apply to see grouped bars. The "Split" category is now shown along the X axis with a colored bar for each value in the "X Axis Categories" selection chosen earlier. A legend shows the color map.
Further customize your visualization using the Chart Type and Chart Layout links in the upper right.
Chart Type reopens the creation dialog allowing you to:
Change the "X Axis Categories" column (hover and click the X to delete the current election).
Remove or change the Y Axis metric, the "Split Categories By" column, or the aggregation method.
You can also drag and drop columns between selection boxes to change how each is used.
Note that you can also click another chart type on the left to switch how you visualize the data with the same axes when practical.
Click Apply to update the chart with the selected changes.
Change Layout
Chart Layout offers the ability to change the look and feel of your chart.
There are 3 tabs:
General:
Provide a Title to show above your chart. By default, the dataset name is used; at any time you can return to this default by clicking the (refresh) icon for the field.
Provide a Subtitle to print under the chart title.
Specify the width and height.
You can also customize the opacity, line width, and line color for the bars.
Select one of three palettes for bar fill colors: Light, Dark, or Alternate. The array of colors is shown.
Margins (px): If the default chart margins cause axis labels to overlap, or you want to adjust them for other reasons, you can specify them explicitly in pixels. Specify any one or all of the top, bottom, left, and right margins. See an example here.
X-Axis/Y-Axis:
Label: Change the display labels for the axis (notice this does not change which column provides the data). Click the icon to restore the original label based on the column name.
For the Y-axis, the Range shown can also be specified - the default is Automatic across charts. Select Automatic Within Chart to have the range based only on this chart. You can also select Manual and specify the min and max values directly.
Click Apply to update the chart with the selected changes.
Save and Export Charts
When your chart is ready, click Save.
Name the chart, enter a description (optional), and choose whether to make it viewable by others. You will also see the default thumbnail which has been auto-generated, and can choose whether to use it. As with other charts, you can later attach a custom thumbnail if desired.
Once you have created a bar chart, it will appear in the Data Browser and on the (charts) menu for the source dataset. You can manage metadata about it as described in Manage Data Views.
Export Chart
Hover over the chart to reveal export option buttons in the upper right corner:Export your completed chart by clicking an option:
PDF: generate a PDF file.
PNG: create a PNG image.
Script: pop up a window displaying the JavaScript for the chart which you can then copy and paste into a wiki. See Tutorial: Visualizations in JavaScript for a tutorial on this feature.
A box plot, or box-and-whisker plot, is a graphical representation of the range of variability for a measurement. The central quartiles (25% to 75% of the full range of values) are shown as a box, and there are line extensions ('whiskers') representing the outer quartiles. Outlying values are typically shown as individual points.
Select the column to use on the Y axis and drag it to the Y Axis box.
Only the Y Axis field is required to create a basic single-box plot, but there are additional options.
Select another column and choose how to use this column:
X Axis Categories: Create a plot with multiple boxes along the x-axis, one per value in the selected column.
Color: Display values in the plot with a different color for each column value. Useful when displaying all points or displaying outliers as points.
Shape: Change the shape of points based on the value in the selected column. 5 different shapes are available.
Here we make it the X-Axis Category and click Apply to see a box plot for each cohort.
Click View Data to see, filter, or export the underlying data.
Click View Chart to return. If you applied any filters, you would see them immediately reflected in the plot.
Box Plot Customizations
Customize your visualization using the Chart Type and Chart Layout links in the upper right.
Chart Type reopens the creation dialog allowing you to:
Change any column selection (hover and click the X to delete the current election). You can also drag and drop columns between selection boxes to change positions.
Add new columns, such as to group points by color and shape. Don't forget to change the layout as described below to fully see these changes.
Click Apply to see your changes and switch dialogs.
Chart Layout offers options to change the look of your chart, including these changes to make our color and shape distinctions clearer:
Set Show Points to AllAND:
Check Jitter Points to spread the points out horizontally.
Click Apply to update the chart with the selected changes.
Below we see a plot with all data shown as points, jittered to spread them out and show the different colors and shapes of points. Notice the legend in the upper right. Hover over any point for details about it.
You may also notice that the outline of the overall box plot has not changed from the basic fill version shown above. This enhanced chart is giving additional information without losing the big picture of the basic plot.
Change Layout
Chart Layout offers the ability to change the look and feel of your chart.
There are 4 tabs:
General:
Provide a Title to show above your plot. By default, the dataset name is used, and you can return to this default at an time by clicking the refresh icon.
Provide a Subtitle to show below the title.
Specify the width and height.
Elect whether to display single points for all data, only for outliers, or not at all.
Check the box to jitter points.
You can also customize the colors, opacity, width and fill for points or lines.
Margins (px): If the default chart margins cause axis labels to overlap, or you want to adjust them for other reasons, you can specify them explicitly in pixels. Specify any one or all of the top, bottom, left, and right margins. See an example here.
X-Axis:
Label: Change the display label for the X axis (notice this does not change which column provides the data). Click the icon to restore the original label based on the column name.
Y-Axis:
Label: Change the display label for the Y axis as for the X axis.
Scale Type: Choose log or linear scale for the Y axis.
Range: Let the range be determined automatically or specify a manual range (min/max values) for the Y axis.
Developer: Only available to users that have the "Platform Developers" site role.
A developer can provide a JavaScript function that will be called when a data point in the chart is clicked.
Provide Source and click Enable to enable it.
Click the Help tab for more information on the parameters available to such a function.
Click Apply to update the chart with the selected changes.
Save and Export Plots
When your chart is ready, click Save.
Name the plot, enter a description (optional), and choose whether to make it viewable by others. You will also see the default thumbnail which has been auto-generated. You can elect None. As with other charts, you can later attach a custom thumbnail if desired.
Once you have created a box plot, it will appear in the Data Browser and on the (charts) menu for the source dataset. You can manage metadata about it as described in Manage Data Views.
Export Chart
Hover over the chart to reveal export option buttons in the upper right corner:Export your completed chart by clicking an option:
PDF: generate a PDF file.
PNG: create a PNG image.
Script: pop up a window displaying the JavaScript for the chart which you can then copy and paste into a wiki. See Tutorial: Visualizations in JavaScript for a tutorial on this feature.
Rules Used to Render the Box Plot
The following rules are used to render the box plot. Hover over a box to see a pop-up.
Min/Max are the highest and lowest data points still within 1.5 of the interquartile range.
Q1 marks the lower quartile boundary.
Q2 marks the median.
Q3 marks the upper quartile boundary.
Values outside of the range are considered outliers and are rendered as dots by default. The options and grouping menus offer you control of whether and how single dots are shown.
Developer Extensions
Developers (users with the "Platform Developers" role) can extend plots that display points to run a JavaScript function when a point is clicked. For example, it might show a widget of additional information about the specific data that point represents. Supported for box plots, scatter plots, line plots, and time charts.To use this function, open the Chart Layout editor and click the Developer tab. Provide Source in the window provided, click Enable, then click Save to close the panel.Click the Help tab to see the following information on the parameters available to such a function.Your code should define a single function to be called when a data point in the chart is clicked. The function will be called with the following parameters:
data: the set of data values for the selected data point. Example:
A line plot tracks the value of a measurement across a horizontal scale, typically time. It can be used to show trends alone and relative to other individuals or groups in one plot.
Select the X Axis column by drag and drop, here "Date".
Select the Y Axis column by drag and drop, here "White Blood Count".
Leave the other fields unset/at their defaults for now.
Only the X and Y Axes are required to create a basic line plot. Other options will be explored below.
Click Apply to see the basic plot.
This basic line chart plots a point for every "Y-axis" value measured for each "X-axis" value, as in a scatter plot, then draws a line between them. When all values for all participants are mixed, this data isn't necessarily useful. Next, we might want to separate by participant to see if any patterns emerge for individuals.You may also notice the labels for tickmarks along the X axis overlap the "Date" label. We will fix that below after making other plot changes.
Line Plot Customizations
Customize your visualization using the Chart Type and Chart Layout links in the upper right.
Chart Type reopens the creation dialog allowing you to:
Change the X or Y Axis column (hover and click the X to delete the current selection).
Select a Series column (optional). The series measure is used to split the data into one line per distinct value in the column.
Users of Premium Editions can change the type of Trendline if desired. Learn more below.
Note that you can also click another chart type on the left to switch how you visualize the data with the same axes when practical.
For this walkthrough, drag "Participant ID" to the Series box.
Click Apply.
Now the plot draws series' lines between values for the same subject, but is unusably dense. Let's filter to a subset of interest.
Click View Data to see and filter the underlying data.
Click the ParticipantID column header and select Filter.
Click the "All" checkbox in the popup to unselect all values. Then, check the boxes for the first 3 participants.
Click OK.
Click View Chart to return. Now there are 3 lines showing values for the 3 participants.
Add a Second Y Axis
To plot more data, you can add a second Y axis and display it on the right.
Click Chart Type to reopen the editor.
Drag the "CD4" column to the Y Axis box. Notice it becomes a second panel and does not replace the prior selection (Lymphs).
Click the (circle arrow) to set the Y Axis Side for this measure to be on the right.
Click Apply.
You can see the trend line for each measure for each cohort in a single plot.
Change Chart Layout
The Chart Layout button offers the ability to change the look and feel of your chart.There are four tabs:
General:
Provide a title to display on the plot. The default is the name of the source data grid.
Provide a subtitle to display under the title.
Specify a width and height.
Control the point size and opacity, as well as choose the default color, if no "Series" column is set.
Control the line width.
Hide Data Points: Check this box to display a simple line instead of showing shaped points for each value.
Number of Charts: Select whether to show a single chart, or a chart per measure, when multiple measures are defined.
Margins (px): If the default chart margins cause axis labels to overlap, or you want to adjust them for other reasons, you can specify them explicitly in pixels. Specify any one or all of the top, bottom, left, and right margins here.
X-Axis:
Label: Change the display label for the X axis (notice this does not change which column provides the data). Click the icon to restore the original label based on the column name.
Y-Axis:
Label: Change the display label for the Y axis as for the X axis.
Scale Type: Choose log or linear scale for the Y axis.
Range: For the Y-axis, the default is Automatic across charts. Select Automatic Within Chart to have the range based only on this chart. You can also select Manual and specify the min and max values directly.
Developer: Only available to users that have the "Platform Developers" site role.
A developer can provide a JavaScript function that will be called when a data point in the chart is clicked.
Provide Source and click Enable to enable it.
Click the Help tab for more information on the parameters available to such a function.
When there are enough values on an axis that the values overlap the label, or if you want to adjust the margins of your chart for any reason, you can use the chart layout settings to set them. In our example, the date display is too long for the default margin (and overlaps the label) so before publishing, we can improve the look.
Observe the example chart where the date displays overlap the label "Date".
Open the chart for editing, then click Chart Layout.
Scroll down and set the bottom margin to 85 in this example.
You can also adjust the other margins as needed.
Note that plot defaults and the length of labels can both vary, so the specific setting your plot will need may not be 85.
Click Apply.
Click Save to save with the revised margin settings.
Trendline Options (Premium Feature)
The trendline shown in a line plot defaults to being point-to-point, and is adjustable to other options in some situations. Trendline options are available in Premium Editions when creating a line plot in the LabKey Server interface, and conditionally available in the LabKey LIMS and Biologics LIMS products when a numeric field is selected for the X axis.Reopen the Chart Type editor to see if the option is available and select the desired trendline if so. Any data can use the first three basic types. The non-linear trendline options are only available for tables and queries in the assay schema.
Point-to-Point (default)
Linear Regression
Polynomial
Nonlinear 3PL
Nonlinear 3PL (Alternate)
Nonlinear 4PL
Nonlinear 4PL (Alternate)
Nonlinear 5PL
The same data presented with four different trendline options:
When a trendline type is selected it will apply to each distinct series, or all data points if no series variable is selected.
Nonlinear trendline options will conditionally show asymptote min/max inputs, when available.
Hovering over a trendline will show the stats and curve fit parameters.
Saving a line chart with a trendline type set will retain that selection and show on render of the chart.
Save and Export Plots
When your plot is finished, click Save.
Name the chart, enter a description (optional), and choose whether to make it viewable by others. You will also see the default thumbnail which has been auto-generated, and can choose whether to use it. As with other charts, you can later attach a custom thumbnail if desired.
Once you have saved a line plot, it will appear in the Data Browser and on the (charts) menu for the source dataset. You can manage metadata about it as described in Manage Data Views.
Export Chart
Hover over the chart to reveal export option buttons in the upper right corner:Export your completed chart by clicking an option:
PDF: generate a PDF file.
PNG: create a PNG image.
Script: pop up a window displaying the JavaScript for the chart which you can then copy and paste into a wiki. See Tutorial: Visualizations in JavaScript for a tutorial on this feature.
Select the column to visualize and drag it to the Categories box.
Click Apply. The size of the pie wedges will reflect the count of rows for each unique value in the column selected.
Click View Data to see, filter, or export the underlying data.
Click View Chart to return. If you applied any filters, you would see them immediately reflected in the chart.
Pie Chart Customizations
Customize your visualization using the Chart Type and Chart Layout links in the upper right.
Chart Type reopens the creation dialog allowing you to:
Change the Categories column selection.
Note that you can also click another chart type on the left to switch how you visualize the data using the same selected columns when practical.
Click Apply to update the chart with the selected changes.
Change Layout
Chart Layout offers the ability to change the look and feel of your chart.
Customize any or all of the following options:
Provide a Title to show above your chart. By default, the dataset name is used.
Provide a Subtitle. By default, the categories column name is used. Note that changing this label does not change which column is used for wedge categories.
Specify the width and height.
Select a color palette. Options include Light, Dark, and Alternate. Mini squares showing the selected palette are displayed.
Customizing the radii of the pie chart allows you to size the graph and if desired, include a hollow center space.
Elect whether to show percentages within the wedges, the display color for them, and whether to hide those annotations when wedges are narrow. The default is to hide percentages when they are under 5%.
Use the Gradient % slider and color to create a shaded look.
Click Apply to update the chart with the selected changes.
Save and Export Charts
When your chart is ready, click Save.
Name the chart, enter a description (optional), and choose whether to make it viewable by others. You will also see the default thumbnail which has been auto-generated, and can choose whether to use it. As with other charts, you can later attach a custom thumbnail if desired.
Once you have created a pie chart, it will appear in the Data Browser and on the (charts) menu for the source dataset. You can manage metadata about it as described in Manage Data Views.
Export Chart
Hover over the chart to reveal export option buttons in the upper right corner:Export your completed chart by clicking an option:
PDF: generate a PDF file.
PNG: create a PNG image.
Script: pop up a window displaying the JavaScript for the chart which you can then copy and paste into a wiki. See Tutorial: Visualizations in JavaScript for a tutorial on this feature.
Scatter plots represent the relationship between two different numeric measurements. Each dot is positioned based on the value of the values selected for the X and Y axes.
Optionally select columns for grouping of points by color or shape.
Note that you can also click another chart type on the left to switch how you visualize the data with the same axes and color/shape groupings when practical.
Click Apply to update the chart with the selected changes.
Here we see the same scatter plot data, with colors varying by cohort and points shaped based on treatment group. Notice the key in the upper right.
Change Layout
The Chart Layout button offers the ability to change the look and feel of your chart.There are four tabs:
General:
Provide a title to display on the plot. The default is the name of the source data grid.
Provide a subtitle to display under the title.
Specify a width and height.
Choose whether to jitter points.
Control the point size and opacity, as well as choose the default color palette. Options: Light (default), Dark, and Alternate. The array of colors is shown under the selection.
Number of Charts: Select either "One Chart" or "One Per Measure".
Group By Density: Select either "Always" or "When number of data points exceeds 10,000."
Grouped Data Shape: Choose either hexagons or squares.
Density Color Palette: Options are blue & white, heat (yellow/orange/red), or select a single color from the dropdown to show in graded levels. These palettes override the default color palette and other point options in the left column.
Margins (px): If the default chart margins cause axis labels to overlap, or you want to adjust them for other reasons, you can specify them explicitly in pixels. Specify any one or all of the top, bottom, left, and right margins. See an example here.
X-Axis/Y-Axis:
Label: Change the display labels for the axis (notice this does not change which column provides the data). Click the icon to restore the original label based on the column name.
Scale Type: Choose log or linear scale for each axis.
Range: Let the range be determined automatically or specify a manual range (min/max values) for each axis.
Developer: Only available to users that have the "Platform Developers" site role.
A developer can provide a JavaScript function that will be called when a data point in the chart is clicked.
Provide Source and click Enable to enable it.
Click the Help tab for more information on the parameters available to such a function.
You can add more data to a scatter plot by selecting a second Y axis column. Reopen a chart for editing, click Chart Type, then drag another column to the Y Axis field. The two selected fields will both have panels. On each you can select the side for the Y Axis using the arrow icons.For this example, we've removed the color and shape columns to make it easier to see the two axes in the plot. Click Apply.If you use the Chart View > Number of Charts > One Per Measure, you will see two separate charts, still respecting the Y Axis sides you set.
Example: Heat Map
Displaying a scatter plot as a heatmap is done by changing the layout of a chart. Very large datasets are easier to interpret as heatmaps, grouped by density (also known as point binning).
Click Chart Layout and change Group By Density to "Always".
Select Heat as the Density Color Palette and leave the default Hexagon shape selected
Click Apply to update the chart with the selected changes. Shown here, the number of charts was reset to one, and only a single Y axis is included.
Notice that when binning is active, a warning message will appear reading: "The number of individual points exceeds XX. The data is now grouped by density which overrides some layout options." XX will be either 10,000 or 1, if you selected "Always" as we did. Click Dismiss to remove that message from the plot display.
Save and Export Plots
When your plot is finished, click Save.
Name the plot, enter a description (optional), and choose whether to make it viewable by others. You will also see the default thumbnail which has been auto-generated, and can choose whether to use it. As with other charts, you can later attach a custom thumbnail if desired.
Once you have saved a scatter plot, it will appear in the Data Browser and on the (charts) menu for the source dataset. You can manage metadata about it as described in Manage Data Views.
Export Plot
Hover over the chart to reveal export option buttons in the upper right corner:Export your completed chart by clicking an option:
PDF: generate a PDF file.
PNG: create a PNG image.
Script: pop up a window displaying the JavaScript for the chart which you can then copy and paste into a wiki. See Tutorial: Visualizations in JavaScript for a tutorial on this feature.
Time charts provide rich time-based visualizations for datasets and are available in LabKey study folders. In a time chart, the X-axis shows a calculated time interval or visit series, while the Y-axis shows one or more numerical measures of your choice.
Individually select which study participants, cohorts, or groups appear in the chart.
Refine your chart by defining data dimensions and groupings.
Export an image of your chart to a PDF or PNG file.
Export your chart to Javascript (for developers only).
Note: In a visit-based study, visits are a way of measuring sequential data gathering. To create a time chart of visit based data, you must first create an explicit ordering of visits in your study. Time charts are not supported for continuous studies, because they contain no calculated visits/intervals.
Create a Time Chart
Navigate to the dataset, view, or query of interest in your study. In this example, we use the Lab Results dataset in the example study.
Select (Charts) > Create Chart. Click Time.
Whether the X-axis is date based or visit-based is determined by the study type. For a date-based study:
Choose the Time Interval to plot: Days, Weeks, Months, Years.
Select the desired Interval Start Date from the pulldown menu. All eligible date fields are listed.
At the top of the right panel is a drop down from which you select the desired dataset or query. Time charts are only supported for datasets/queries in the "study" schema which include columns designated as 'measures' for plotting. Queries must also include both the 'ParticipantId' and 'ParticipantVisit' columns to be listed here.
The list of columns designated as measures available in the selected dataset or query is shown in the Columns panel. Drag the desired selection to the Y-Axis box.
By default the axis will be shown on the left; click the right arrow to switch sides.
Click Apply.
The time chart will be displayed.
Use the checkboxes in the Filters panel on the left:
Click a label to select only that participant.
Click a checkbox to add or remove that participant from the chart.
Click View Data to see the underlying data.
Click View Chart(s) to return.
Time Chart Customizations
Customize your visualization using the Chart Type and Chart Layout links in the upper right.
Chart Type reopens the creation dialog allowing you to:
Change the X Axis options for time interval and start date.
Change the Y Axis to plot a different measure, or plot multiple measures at once. Time charts are unique in allowing cross-query plotting. You can select measures from different datasets or queries within the same study to show on the same time chart.
Remove the existing selection by hovering and clicking the X. Replace with another measure.
Add a second measure by dragging another column from the list into the Y-Axis box.
For each measure you can specify whether to show the Y-axis for it on the left or right.
Open and close information panels about time chart measures by clicking on them.
Click Apply to update the chart with the selected changes.
Change Layout
Chart Layout offers the ability to change the look and feel of your chart.
There are at least 4 tabs:
On the General tab:
Provide a Title to show above your chart. By default, the dataset name is used.
Specify the width and height of the plot.
Use the slider to customize the Line Width.
Check the boxes if you want to either Hide Trend Line or Hide Data Points to get the appearance you prefer. When you check either box, the other option becomes unavailable.
Number of Charts: Choose whether to show all data on one chart, or separate by group, or by measure.
Subject Selection: By default, you select participants from the filter panel. Select Participant Groups to enable charting of data by groups and cohorts using the same checkbox filter panel. Choose at least one charting option for groups:
Show Individual Lines: show plot lines for individual participant members of the selected groups.
Show Mean: plot the mean value for each participant group selected. Use the pull down to select whether to include range bars when showing mean. Options are: "None, Std Dev, or Std Err".
On the X-Axis tab:
Label: Change the display label shown on the X-axis. Note that changing this text will not change the interval or range plotted. Use the Chart Type settings to change what is plotted.
Range: Let the range be determined automatically or specify a manual range (min/max values).
There will be one Y-Axis tab for each side of the plot if you have elected to use both the left and right Y-axes. For each side:
Label Change the display labels for this Y-axis. Note that changing this text will not change what is plotted. Click the icon to restore the original label based on the column name.
Scale Type: Choose log or linear scale for each axis.
Range: Let the range be determined automatically or specify a manual range (min/max values) for each axis.
For each Measure using that Y-axis:
Choose an Interval End Date. The pulldown menu includes eligible date columns from the source dataset or query.
Choose a column if you want to Divide Data Into Series by another measure.
When dividing data into series, choose how to display duplicate values (AVG, COUNT, MAX, MIN, or SUM).
Developer: Only available to users that have the "Platform Developers" site role.
A developer can provide a JavaScript function that will be called when a data point in the chart is clicked.
Provide Source and click Enable to enable it.
Click the Help tab for more information on the parameters available to such a function.
Click Apply to update the chart with the selected changes. In this example, we now plot data by participant group. Note that the filter panel now allows you to plot trends for cohorts and other groups. This example shows a plot combining trends for two measures, lymphs and viral load, for two study cohorts.
Save Chart
When your chart is ready, click Save.
Name the chart, enter a description (optional), and choose whether to make it viewable by others. You will also see the default thumbnail which has been auto-generated, and can choose whether to use it. As with other charts, you can later attach a custom thumbnail if desired.
Click Save.
Once you have created a time chart, it will appear in the Data Browser and on the charts menu for the source dataset.
Data Dimensions
By adding dimensions for a selected measure, you can further refine the timechart. You can group data for a measure on any column in your dataset that is defined as a "data dimension". To define a column as a data dimension:
Open a grid view of the dataset of interest.
Click Manage.
Click Edit Definition.
Click the Fields section to open it.
Expand the field of interest.
Click Advanced Settings.
Place a checkmark next to Make this field available as a dimension.
Click Apply.
Click Save.
To use the data dimension in a time chart:
Click View Data to return to your grid view.
Create a new time chart, or select one from the (Charts) menu and click Edit.
Click Chart Layout.
Select the Y-Axis tab for the side of the plot you are interested in (if both are present.
The pulldown menu for Divide Data Into Series By will include the dimensions you have defined.
Select how you would like duplicate values displayed. Options: Average, Count, Max, Min, Sum.
Click Apply.
A new section appears in the filters panel where you can select specific values of the new data dimension to further refine your chart.
Export Chart
Hover over the chart to reveal export option buttons in the upper right corner:Export your completed chart by clicking an option:
PDF: generate a PDF file.
PNG: create a PNG image.
Script: pop up a window displaying the JavaScript for the chart which you can then copy and paste into a wiki. See Tutorial: Visualizations in JavaScript for a tutorial on this feature.
Click a column header to see a list of Column Visualizations, small plots and charts that apply to a single column. When selected, the visualization is added to the top of the data grid. Several can be added at a time, and they are included within a saved custom grid view. When you come back to the saved view, the Column Visualizations will appear again.
Visualizations are always 'live', reflecting updates to the underlying data and any filters added to the data grid.To remove a chart, hover over the chart and click the 'X' in the upper right corner.Available visualization types are determined by datatype as well as whether the column is a Measure and/or a Dimension.
The box plot option is shown for any column marked as a Measure.
The bar and pie chart options are shown for any column marked as a Dimension.
Column visualizations are simplified versions of standalone charts of the same types. Click any chart to open it within the plot editor which allows you to make many additional customizations and save it as a new standalone chart.
Bar Chart
A histogram of the Weight column.
Box and Whisker Plot
A basic box plot report. You can include several column visualizations above a grid simultaneously.
Pie Chart
A pie chart showing prevalence of ARV Regimen types.Filters are also applied to the visualizations displayed. If you filter to exclude 'blank' ARV treatment types, the pie chart will update.
Quick Charts provide a quick way to assess your data without deciding first what type of visualization you will use. A best guess visualization for the data in a single column is generated and can be refined from there.
Depending on the content of the column, LabKey Server makes a best guess at the type and arrangement of chart to use as a starting place. A numeric column in a cohort study, for example, might be quickly charted as a box and whisker plot using cohorts as categories.
Refine the Chart
You can then alter and refine the chart in the following ways:
View Data: Toggle to the data grid, potentially to apply filters to the underlying data. Filters are reflected in the plot upon re-rendering.
Export: Export the chart as a PDF, PNG, or Script.
Help: Documentation links.
Chart Type: Click to open the plot editor. You can change the plot type to any of the following and the options for chart layout settings will update accordingly. In the plot editor, you can also incorporate data from other columns.
Chart Layout: Click to customize the look and feel of your chart; options available vary based on the chart type. See individual chart type pages for a descriptions of options.
Users can integrate their LabKey Server with Tableau Desktop via an Open Database Connectivity (ODBC) integration. Data stored in LabKey can be dynamically queried directly from the Tableau application to create reports and visualizations.
Configure LabKey
LabKey must first be configured to accept external analytics integrations using an ODBC connection. At all times, LabKey security settings govern who can see which data; any user must have at least the Reader role to access data from Tableau.Learn more about setting up the ODBC connection in LabKey in these topics:
Within Tableau, once you have loaded the data table you wish to use, you can see the available measures and dimensions listed. Incorporate them into the visualizations by dragging and dropping. Instructions for using Tableau Desktop can be found on their website.In this example, the Physical Exam and Demographics joined view is being used to create a plot showing the CD4 levels over time for two Treatment Groups, those treated and not treated with ARV regimens.Tableau also offers the ability to easily create trend lines. Here an otherwise cluttered scatter plot is made clearer using trend lines:You can build a wide variety of visualizations and dashboards in Tableau.
Video Demonstration
This video covers the ways in which LabKey Server and Tableau Desktop are great partners for creating powerful visualizations.
A List is a user-defined table that can be used for a variety of purposes:
As a data analysis tool for spreadsheet data and other tabular-format files, such as TSVs and CSVs.
As a place to store and edit data entered by users via forms or editable grids
To define vocabularies, which can be used to constrain choices during completion of fields in data entry forms
As read-only resources that users can search, filter, sort, and export
The design, or schema, of a list is the set of fields (columns and types), including the identification of the primary key. Lists can be linked via lookups and joins to draw data from many sources. Lists can be indexed for search, including optional indexing of any attachments added to fields. Populated lists can be exported and imported as archives for easy transfer between folders or servers.
You need to be an administrator to create and manage lists. You can directly access the list manager by selecting > Manage Lists. To make the set of lists visible to other users, and create a one click shortcut for admins to manage lists, add a Lists web part to your project or folder.
Lists Web Part
Enter > Page Admin Mode.
Choose Lists from the <Select Web Part> pulldown at the bottom of the page.
Click Add.
Click Exit Admin Mode.
List-Single Web Part
To display the contents of a single list, add a List - Single web part, name it and choose the list and view to display.
Tutorial: Lists
This tutorial introduces you to Lists, the simplest way to represent tabular data in LabKey. While straightforward, lists support many advanced tools for data analysis. Here you will learn about the power of lookups, joins, and URL properties for generating insights into your results.
Lists can be simple literal one column "lists" or many columned tables with a set of values for each "item" on the list. A list must always have a unique primary key - if your data doesn't have one naturally, you can use an auto-incrementing integer to guarantee uniqueness.Lists offer many data connection and analysis opportunities we'll begin to explore in this tutorial:
Lookups can help you display names instead of code numbers, present options to users adding data, and interconnect tables.
Joins help you view and present data from related lists together in shared grids without duplicating information.
URL properties can be used to filter a grid view of a list OR help you link directly to information stored elsewhere.
Completing this tutorial as written requires administrative permissions, which you will have if you create your own server trial instance in the first step. The features covered are not limited to admin users.
You'll see the set of fields that define the columns in your list.
In the blue banner, you must select a field to use as the primary key. Using the name could be non-unique, and even badge numbers might be reassigned, so select Auto integer key from the dropdown and notice that a "Key" field is added to your list.
Click Save.
You'll see the empty "frame" for your new list with column names but "No data to show."
Now the "Technicians" list will appear in the Lists web part when you return to the main folder page.
Populate a List
You can populate a list one row at a time or in bulk by importing a file (or copying and pasting the data).
If you returned to the List Tutorial main page, click the Technicians list name to reopen the empty frame.
Click (Insert data) and select Insert new row.
Enter your own name, make up a "Department" and use any number as your badge number.
Click the Upload file panel to open it, then click Browse (or Choose File) and choose the "Technicians.xls" file you downloaded.
Alternately, you could open the file and copy paste the contents (including the headers) into the copy/paste text panel instead.
Click Submit.
You will now see a list of some "Technicians" with their department names. Before we continue, let's add a few more lists to the folder using a different list creation option below.
Import a List Archive
A list archive can be exported from existing LabKey lists in a folder, or manually constructed following the format. It provides a bulk method for creating and populating sets of lists.
Click here to download the Tutorial.lists.zip archive. Do not unzip it.
Click Choose File and select Tutorial.lists.zip from where you downloaded it.
Click Import List Archive.
Now you will see several additional lists have been added to your folder. Click the names to review, and continue the tutorial using this set of lists.
You can interconnect lists with each other by constructing "lookups", then use those connections to present joined grids of data within the user interface.
Connect Lists with a Lookup
Click the Technicians list from the Lists web part.
The grid shows the list of technicians and departments, including yourself. You can see the Department column displays a text name, but is currently unlinked. We can connect it to the "Departments" list uploaded in the archive. Don't worry that the department you typed is probably not on that list.
Click Design to open the list design for editing.
Click the Fields section to open it.
For the Department field, click the Text selector and choose the Data Type Lookup.
The details panel for the field will expand. Under Lookup Definition Options:
Leave the Target Folder set to the current folder.
Set the Target Schema to "lists".
On the Target Table dropdown, select "Departments (String)".
Click Save.
Now the list shows the values in the Department column as links. If you entered something not on the list, it will not be linked, but instead be plain text surrounded by brackets.Click one of the links to see the "looked up" value from the Departments list. Here you will see the fields from the "Departments" list: The contact name and phone number.
You may have noticed that while there are several lists in this folder now, you did not see them all on the dropdown for setting the lookup target. The field was previously a "Text" type field, containing some text values, so only lists with a primary key of that "Text" type are eligible to be the target when we change it to be a lookup.
Create a Joined Grid
What if you want to present the details without your user having to click through? You can easily "join" these two lists as follows.
Select > Manage Lists.
Click Technicians.
Select (Grid Views) > Customize Grid.
In the Available Fields panel, notice the field Department now shows an (expansion) icon. Click it.
Place checkmarks next to the Contact Name and Contact Phone fields (to add them to the Selected Fields panel).
Click View Grid.
Now you see two additional columns in the grid view.
Above the grid, click Save.
In the Save Custom Grid View popup menu, select Named and name this view DepartmentJoinedView.
Click Save in the popup.
You can switch between the default view of the Technicians list and this new joined grid on the (Grid Views) menu.
Use the Lookup to Assist Input
Another reason you might use a lookup field is to help your users enter data.
Hover over the row you created first (with your own name).
Click the (pencil) icon that will appear.
Notice that instead of the original text box, the entry for "Department" is now done with a dropdown.
You may also notice that you are only editing fields for the local list - while the grid showed contact fields from the department list, you cannot edit those here.
Select a value and click Submit.
Now you will see the lookup also "brought along" the contact information to be shown in your row.In the next step, we'll explore using the URL attribute of list fields to make more connections.
In this previous step, we used lookups to link our lists to each other. In this step, we explore two ways to use the URL property of list fields to create other links that might be useful to researchers using the data.
It can be handy to generate an active filtering link in a list (or any grid of data). For example, here we use a URL property to turn the values in the Department column into links to a filtered subset of the data. When you click one value, you get a grid showing only rows where that column has the same value.
If you navigated away from the Technicians list, reopen it.
When you click a value in the "Department" column, notice that you currently go to the contact details. Go back to the Technicians list.
Click the column header Department to open the menu, then select Filter....
Click the label Executive to select only that single value.
Click OK to see the subset of rows.
Notice the URL in your browser, which might look something like this - the full path and your list ID number may vary, but the filter you applied (Department = Executive) is encoded at the end.
Clear the filter using the in the filter bar, or by clicking the column header Department and then clicking Clear Filter.
Now we'll modify the design of the list to turn the values in the Department column into custom links that filter to just the rows for the value that we click.
This URL starts with "/" indicating it is local to this container.
The filter portion of this URL replaces "Executive" with the substitution string "${Department}", meaning the value of the Department column. (If we were to specify "Executive", clicking any Department link in the list would filter the list to only show the executives!)
The "listId" portion of the URL has been replaced with "name=Technicians." This allows the URL to work even if exported to another container where the listId might be different.
Scroll down and click Save.
Now notice that when you click a value in the "Department" column, you get the filtering behavior we just defined.
Click Documentation in any row and you will see the list filtered to display only rows for that value.
Learn more about ways to customize URLs in this topic: URL Field Property
Create Links to Outside Resources
A column value can also become a link to a resource outside the list, and even outside the server. All the values in a column could link to a fixed destination (such as to a protocol document or company web page) or you can make row-specific links to files where a portion of the link URL matches a value in the row such as the Badge Number in this example.For this example, we've stored some images on our support site, so that you can try out syntax for using both a full URL to reference a non-local destination AND the use of a field value in the URL. In this case, images are stored as <badge number>.png; in actual use you might have locally stored slide images or other files of interest named by subjectId or another column in your data.Open this link in a new browser window:
You can directly edit the URL, substituting the other badge IDs used in the Technicians list you loaded (701, 1802, etc) where you see 104 in the above.Here is a generalized version, using substitution syntax for the URL property, for use in the list design.
Click the List Tutorial link near the top of the page, then click the Technicians list in the Lists web part.
Click Design.
Click the Fields section to expand it.
Expand the Badge field.
Into the URL property for this field, paste the generalized version of the link shown above.
Click Save.
Observe that clicking one of the Badge Number values will open the image with the same name.
If you edit your own row to set your badge number to "1234" you will have an image as well. Otherwise clicking a value for which there is no pre-loaded image will raise an error.
Congratulations
You've now completed the list tutorial. Learn more about lists and customizing the URL property in the related topics.
A list is a basic way of storing tabular information. LabKey lists are flexible, user-defined tables that can have as many columns as needed. Lists must have a primary key field that ensures rows are uniquely identified.The list design is the set of columns and types, which forms the structure of the list, plus the identification of the primary key field, and properties about the list itself.
In the project or folder where you want the list, select > Manage Lists.
Click Create New List.
Name the list, i.e. "Technicians" in this example.
The name must be unique, must start with a letter or number character, and cannot contain special characters or some reserved substrings listed here.
If the name is taken by another list in the container, project, or /Shared project, you'll see an error and need to choose another name. You may also want to use that shared definition instead of creating a new list definition.
Adding a Description is optional, but can give other users more information about the purpose of this list.
Use the checkboxes to decide whether to Allow these Actions for the list:
Delete
Upload
Export & Print
Continue to define fields and other settings before clicking Save.
Set Advanced Properties
Click Advanced Settings to set more properties (listed below the image).
Default Display Field: Once some fields have been defined, use this dropdown to select the field that should be displayed by default when this list is used as a lookup target.
There is also an AUTO option that will use the first string field or the primary key if there are no string fields.
Search Indexing Options (by default, no options are selected):
Continue to define the list fields before clicking Save.
Define List Fields and Set Primary Key
The fields in the list define which columns will be included. There must be a primary key column to uniquely identify the rows. The key can either be an integer or text field included with your data, OR you can have the system generate an auto-incrementing integer key that will always be unique. Note that if you select the auto-incrementing integer key, you will not have the ability to merge list data.You have three choices for defining fields:
Click Manually Define Fields (under the drag and drop region).
Key Field Name:
If you want to use an automatically incrementing integer key, select Auto integer key. You can rename the default Key field that will be added.
If you want to use a different field (of Integer or Text type), first define the fields, then select from this dropdown.
Use Add Field to add the fields for your list.
Specify the Name and Data Type for each column.
Check the Required box to make providing a value for that field mandatory.
Open a field to define additional properties using the expansion icon.
Remove a field if necessary by clicking the .
Details about adding fields and editing their properties can be found in this topic: Field Editor.
Scroll down and click Save when you are finished.
Infer Fields from a File
Instead of creating the list fields one-by-one you can infer the list design from the column headers of a sample spreadsheet. When you first click the Fields section, the default option is to import or infer fields. Note that inferring fields is only offered during initial list creation and cannot be done when editing a list design later. If you start manually defining fields and decide to infer instead, delete the manually defined fields and you will see the inferral option return. Note that you cannot delete an auto integer key field and will need to restart list creation from scratch if you have manually chosen that key type already.
Name the list, i.e. "Technicians2" so you can compare it to the list created above.
Click the Fields section to open it.
Drag and drop the downloaded "Technicians.xls" file into the target area.
The fields will be inferred and added automatically.
Note that if your file includes columns for reserved fields, they will not be shown as inferred. Reserved fields will always be created for you.
Select the Key Field Name - in this case, select Auto integer key to add a new field to provide our unique key.
If you need to make changes or edit properties of these fields, follow the instructions above or in the topic: Field Editor.
In particular, if your field names include any special characters (including spaces) you should adjust the inferral to give the field a more 'basic' name and move the original name to the Label and Import Aliases field properties. For example, if your data includes a field named "CD4+ (cells/mm3)", you would put that string in both Label and Import Aliases but name the field "CD4" for best results.
By default the contents of the spreadsheet you used for inferral will be imported to this list when you click Save.
If you do not want to do that, uncheck the box.
Scroll down and click Save.
Click "Technicians2" to see that the field names and types are inferred forming the header row, but no data was imported from the spreadsheet.
Export/Import Field Definitions
In the top bar of the list of fields, you see an Export button. You can click to export field definitions in a JSON format file. This file can be used to create the same field definitions in another list, either as is or with changes made offline.To import a JSON file of field definitions, use the infer from file method, selecting the .fields.json file instead of a data-bearing file. Note that importing or inferring fields will overwrite any existing fields; it is intended only for new list creation.You'll find an example of using this option in the first step of the List TutorialLearn more about exporting and importing sets of fields in this topic: Field Editor
Shortcut: Infer Fields and Populate a List from a Spreadsheet
If you want to both infer the fields to design the list and populate the new list with the data from the spreadsheet, follow the inferral of fields process above, but leave the box checked in the Import Data section as shown below. The first three rows are shown in the preview.
Click Save and the entire spreadsheet of data will be imported as the list is created.
Note that data previews do not apply field formatting defined in the list itself. For example, Date and DateTime fields are always shown in ISO format (yyyy-MM-dd hh:mm) regardless of source data or destination list formatting that will be applied after import. Learn more in this topic.
Editing the list design allows you change the structure (columns) and functions (properties) of a list, whether or not it has been populated with data.
To see and edit the list design, click Design above the grid view of the list. You can also select > Manage Lists and click Design next to the list name. If you do not see these options, you do not have permission to edit the given list.
List Properties
The list properties contain metadata about the list and enable various actions including export and search.The properties of the Technicians list in the List Tutorial Demo look like this:
Name: The displayed name of the list.
The name must be unique, must start with a letter or number character, and cannot contain special characters or some reserved substrings listed here.
Description: An optional description of the list.
Allow these Actions: These checkboxes determine what actions are allowed for the list. All are checked by default.
Delete
Upload
Export and Print
Click Advanced Settings to see additional properties in a popup:
Default Display Field: Identifies the field (i.e., the column of data) that is used when other lists or datasets do lookups into this list. You can think of this as the "lookup display column." Select a specific column from the dropdown or leave the default "<AUTO>" selection, which will use the first non-lookup string column and if there are no string fields, use the primary key.
Search Indexing Options. Determines how the list data, metadata, and attachments are indexed for full-text searching. Details are below.
List Fields
Click the Fields section to add, delete or edit the fields of your list. Click the to edit details and properties for each field. Learn more in the topic: Field Editor.Example. The field editor for the Technicians list in the List Tutorial Demo looks like this:
Customize the Order of List Fields
By default, the order of fields in the default grid is used to order the fields in insert, edit and details for a list. All fields that are not in the default grid are appended to the end. To see the current order, click Insert New for an existing list.To change the order of fields, drag and drop them in the list field editor using the six-block handle on the left. You can also modify the default grid for viewing columns in different orders. Learn more in the topic: Customize Grid Views.
List Metadata and Hidden Fields
In addition to the fields you define, there is list metadata associated with every list. To see and edit it, use the schema browser. Select > Developer Links > Schema Browser. Click lists and then select the specific list. Click Edit Metadata.List metadata includes the following fields in addition to any user defined fields.
Name
Datatype
Description
Created
date/time
When the list was created
CreatedBy
int (user)
The user who created the list
Modified
date/time
When the list was last modified
ModifiedBy
int (user)
The user who modified the list
container (Folder)
lookup
The folder or project where the list is defined
lastIndexed
date/time
When the list was last indexed
EntityId
text
A unique identifier for this list
There are several built in hidden fields in every list. To see them, open (Grid Views) > Customize Grid and check the box for Show Hidden Fields.
Name
Datatype
Description
Last Indexed
date/time
When this list was last indexed.
Key
int
The key field (if not already shown).
Entity Id
text
The unique identifier for the list itself.
Full-Text Search Indexing
You can control how your list is indexed for search depending on your needs. Choose one or more of the options on the Advanced List Settings popup under the Search Indexing Options section. Clicking the for the first two options adds additional options in the popup:
Index file attachments: Indexes the contents of documents uploaded to attachment fields.
When indexing either the entire list or each item separately, you also specify how to display the title in search results and which fields to index. Note that you may choose to index *both* the entire list and each item, potentially specifying different values for each of these options. When you specify which fields in the list should be indexed, Do not include fields that contain PHI or PII. Full text search results could expose this information.
Index entire list as a single document
Document Title: Any text you want displayed and indexed as the search result title. There are no substitution parameters available for this title. Leave the field blank to use the default title.
Select one option for the metadata/data:
Include both metadata and data: Not recommended for large lists with frequent updates, since updating any item will cause re-indexing of the entire list.
Include data only: Not recommended for large lists with frequent updates, since updating any item will cause re-indexing of the entire list.
Include metadata only (name and description of list and fields). (Default)
Select one option for indexing of PHI (protected health information):
Index all non-PHI text fields
Index all non-PHI fields (text, number, date, and boolean)
Index using custom template: Choose the exact set of fields to index and enter them as a template in the box when you select this option. Use substitution syntax like, for example: ${Department} ${Badge} ${Name}.
Index each item as a separate document
Document Title: Any text you want displayed and indexed as the search result title (ex. ListName - ${Key} ${value}). Leave the field blank to use the default title.
Select one option for indexing of PHI (protected health information):
Index all non-PHI text fields
Index all non-PHI fields (text, number, date, and boolean)
Index using custom template: Choose the exact set of fields to index and enter them as a template in the box when you select this option. Use substitution syntax like, for example: ${Department} ${Badge} ${Name}.
In each case, you can open the list by selecting > Manage Lists and clicking the list name. If your folder includes a Lists web part, you can click the list name directly there.
Insert Single Rows
One option for simple lists is to add a single row at a time:
Select (Insert data) > Insert new row.
Enter the values for each column in the list.
Click Submit.
Import Bulk Data
You can also import multiple rows at once by uploading a file or copy/pasting text. To ensure the format is compatible, particularly for complex lists, you can first download a template, then populate it with your data prior to upload.
Copy/Paste Text
Select (Insert data) > Import bulk data.
Click Download Template to obtain a template for your list that you can fill out.
Because this example list has an incrementing-integer key, you won't see the update and merge options available for some lists.
Copy and paste the spreadsheet contents into the text box, including the header row. Using our "Technicians" list example, you can copy and paste this spreadsheet:
Name
Department
Badge
Abraham Lincoln
Executive
104
Homer
Documentation
701
Marie Curie
Radiology
88
Click Submit.
The pasted rows will be added to the list.
Upload File
Another way to upload data is to directly upload an .xlsx, .xls, .csv, or .txt file containing data.
Again select (Insert data) > Import bulk data.
Using Download Template to create the file you will populate can ensure the format will match.
Click the Upload file (.xlsx, .xls, .csv., .txt) section heading to open it.
Because this example list has an incrementing-integer key, you won't see the update and merge options available for some lists.
Use Browse or Choose File to select the File to Import.
Click Submit.
Update or Merge List Data
If your list has an integer or text key, you have the option to merge list data during bulk import. Both the copy/paste and file import options let you select whether you want to Add rows or Update rows (optionally checking Allow new rows for a data merge).
Update and merge options are not available for lists with an auto-incrementing integer key. These lists always create a new row (and increment the key) during import, so you cannot include matching keys in your imported data.
Import Lookups By Alternate Key
When importing data into a list, either by copy paste or from a file, you can use the checkbox to Import Lookups by Alternate Key. This allows lookup target rows to be resolved by values other than the target's primary key. It will only be available for lookups that are configured with unique column information. For example, tables in the "samples" schema (representing Samples) use the RowId column as their primary key, but their Name column is guaranteed to be unique as well. Imported data can use either the primary key value (RowId) or the unique column value (Name). This is only supported for single-column unique indices. See Add Samples.
View the List
Your list is now populated. You can see the contents of the list by clicking on the name of the list in the Lists web part. An example:Hovering over a row will reveal icon buttons in the first column:
Click the icon to edit a row in a list. You'll see entry fields as when you insert a row, populated with the current values. Make changes as needed and click submit.Changes to lists are audited under List Events in the main audit log. You can also see the history for a specific list by selecting > Manage Lists and clicking View History for the list in question. Learn more here: Manage Lists.
A list is a flexible, user-defined table. To manage all the lists in a given container, an administrator can select > Manage Lists, or click Manage Lists in the Lists web part.
Manage Lists: Manage all the lists in a given folder.
An example list management page from a study folder:
(Grid Views): Customize how this grid of lists is displayed and create custom grid views.
(Charts/Reports): Add a chart or report about the set of lists.
(Delete): Select one or more lists using the checkboxes to activate deletion. Both the data and the list design are removed permanently from your server.
(Export): Export to Excel, Text, Script, or RStudio (when configured).
Export List Archive: Select one or more lists using the checkboxes and export as an archive.
(Print): Print the grid of lists.
Shared List Definitions
The definition of a list can be in a local folder, in the parent project, or in the "/Shared" project. In any given folder, if you select (Grid Views) > Folder Filter, you can choose the set of lists to show. By default, the grid folder filter will show lists in the "Current Folder, Project, and Shared Project".Folder filter options for List definitions:
Current folder
Current folder and subfolders
Current folder, project, and Shared project (Default)
All folders
When you add data to a list, it will be added in the local container, not to any shared location. The definition of a list can be shared, but the data is not, unless you customize the grid view to use a folder filter to expose a wider scope.Folder filter options for List data:
Current folder (Default)
Current folder, subfolders, and Shared project
Current folder, project, and Shared project
All folders
Manage a Specific List
Actions for each list shown in the grid:
Design: Click to view or edit the design, i.e. the set of fields and properties that define the list, including allowable actions and indexing. Learn more in this topic: Edit a List Design.
View History: See a record of all list events and design changes.
Click the Name of the list to see all contents of the list shown as a grid. Options offered for each list include:
(Grid Views): Create custom grid views of this list.
(Charts/Reports): Create charts or reports of the data in this list.
(Insert data): Single row or bulk insert into the list.
(Delete): Select one or more rows to delete.
(Export): Export the list to Excel, text, or script.
Click Design to see and edit the set of fields and properties that define the list.
Click Delete All Rows to empty the data from the list without actually deleting the list structure itself.
(Print): Print the list data.
Add Additional Indices
In addition to the primary key, you can define another field in a list as a key or index, i.e. requiring unique values. Use the field editor Advanced Settings for the field and check the box to Require all values to be unique.
Premium Resource AvailableSubscribers to premium editions of LabKey Server can learn how to add an additional index to a list in this topic:
From the > Manage Lists page, click View History for any list to see a summary of audit events for that particular list. You'll see both:
List Events: Change to the content of the list.
List Design Changes: Changes to the structure of the list.
For changes to data, you will see a Comment "An existing list record was modified". If you hover, then click the (details) link for that row, you will see the details of what was modified.
You can copy some or all of the lists in a folder to another folder or another server using export and import. Exporting a list archive packages up selected lists into a list archive: a .lists.zip file that conforms to the LabKey list export format. The process is similar to study export/import/reload. Information on the list serialization format is covered as part of Study Object Files and Formats.
To export lists in a folder to a list archive, you must have administrator permission to all selected lists. On the Manage Lists page, you may see lists from the parent project as well as the /Shared project.
In the folder that contains lists of interest, select > Manage Lists.
Use the checkboxes to select the lists of interest.
If you want to export all lists in the current container, filter the Folder column to show only the local lists, or use > Folder Filter > Current Folder.
Check the box at the top of the column to select all currently shown lists.
Click Export List Archive.
All selected lists are exported into a zip archive.
Import
To import a list archive:
In the folder where you would like to import the list archive, select > Manage Lists.
Select Import List Archive.
Click Choose File or Browse and select the .zip file that contains your list archive.
Click Import List Archive.
You will see the imported lists included on the Available Lists page.
Note: Existing lists will be replaced by lists in the archive with the same name; this could result in data loss and cannot be undone.
If you exported an archive containing any lists from other containers, such as the parent project or the /Shared project, new local copies of those lists (including their data) will be created when you import the list archive.
Auto-Incrementing Key Considerations
Exporting a list with an auto-increment key may result in different key values on import. If you have lookup lists make sure they use an integer or string key instead of an auto-increment key.
You can leverage the full power of the R statistical programming environment to analyze and visualize datasets on LabKey Server. The results of R scripts can be displayed in LabKey reports that reflect live data updated every time the script is run. Reports may contain text, tables, or charts created using common image formats such as jpeg, png and gif. In addition, the Rlabkey package can be used to insert, update and/or delete data stored on a LabKey Server using R, provided you have sufficient permissions to do so.An administrator must install and configure R on LabKey Server and grant access to users to create and run R scripts on live datasets. Loading of additional packages may also be necessary, as described in the installation topic. Configuration of multiple R engines on a server is possible, but within any folder only a single R engine configuration can be used.
This topic describes how to build reports in the R statistical programming environment to analyze and visualize datasets on LabKey Server. The results of R scripts can be displayed in LabKey reports that reflect live data updated every time the script is run.
Permissions: Creating R Reports requires that the user have both the "Editor" role (or higher) and developer access (one of the roles "Platform Developer" or "Trusted Analyst") in the container. Learn more here: Developer Roles.
Create an R Report from a Data Grid
R reports are ordinarily associated with individual data grids. Choose the dataset of interest and further filter the grid as needed. Only the portion of the dataset visible within this data grid become part of the analyzed dataset.To use the sample dataset we describe in this tutorial, please Tutorial: Set Up a New Study if you have not already done so. Alternately, you may simply add the PhysicalExam.xls demo dataset to an existing study for completing the tutorial. You may also work with your own dataset, in which case steps and screencaps will differ.
View the "Physical Exam" dataset in a LabKey study.
If you want to filter the dataset and thus select a subset or rearrangement of fields, select or create a custom grid view.
Select (Charts/Reports) > Create R Report.
If you do not see the "Create R Report" menu, check to see that R is installed and configured on your LabKey Server. You also need to have the correct permissions to create R Reports. See Configure Scripting Engines for more information.
Create an R Report Independent of any Data Grid
R reports do not necessarily need to be associated with individual data grids. You can also create an R report that is independent of any grid:
Select > Manage Views.
Select Add Report > R Report.
R reports associated with a grid automatically load the grid data into the object "labkey.data". R reports created independently of grids do not have access to labkey.data objects. R reports that pull data from additional tables (other than the associated grid) must use the Rlabkey API to access the other table(s). For details on using Rlabkey, see Rlabkey Package. By default, R reports not associated with a grid are listed under the Uncategorized heading in the list on the Manage Views page.
Review the R Report Builder
The R report builder opens on the Source tab which looks like this. Enter the R script for execution or editing into the Script Source box. Notice the options available below the source entry panel, describe below.
Report Tab
When you select the Report tab, you'll see the resulting graphics and console output for your R report. If the pipeline option is not selected, the script will be run in batch mode on the server.
Data Tab
Select the data tab to see the data on which your R report is based. This can be a helpful resource as you write or refine your script.
Source Tab
When your script is complete and report is satisfactory, return to the Source tab, scroll down, and click Save to save both the script and the report you generated.A saved report will look similar to the results in the design view tab, minus the help text. Reports are saved on the LabKey Server, not on your local file system. They can be accessed through the Reports drop-down menu on the grid view of you dataset, or directly from the Data Views web part.The script used to create a saved report becomes available to source() in future scripts. Saved scripts are listed under the “Shared Scripts” section of the LabKey R report builder.
Help Tab
This Syntax Reference list provides a quick summary of the substitution parameters for LabKey R. See Input/Output Substitutions Reference for further details.
Additional Options
On the Source Tab you can expand additional option sections. Not all options are available to all users, based on permission roles granted.Options
Make this report available to all users: Enables other users to see your R report and source() its associated script if they have sufficient permissions. Only those with read privileges to the dataset can see your new report based on it.
Show source tab to all users: This option is available if the report itself is shared.
Make this report available in child folders: Make your report available in data grids in child folders where the schema and table are the same as this data grid.
Run this report in the background as a pipeline job: Execute your script asynchronously using LabKey's Pipeline module. If you have a big job, running it on a background thread will allow you to continue interacting with your server during execution.
If you choose the asynchronous option, you can see the status of your R report in the pipeline. Once you save your R report, you will be returned to the original data grid. From the Reports drop-down menu, select the report you just saved. This will bring up a page that shows the status of all pending pipeline jobs. Once your report finishes processing, you can click on “COMPLETE” next to your job. On the next page you’ll see "Job Status." Click on Data to see your report.Note that reports are always generated from live data by re-running their associated scripts. This makes it particularly important to run computationally intensive scripts as pipeline jobs when their associated reports are regenerated often.Knitr Options
Select None, HTML, or Markdown processing of HTML source
For Markdown, you can also opt to Use advanced rmarkdown output_options.
Check the box to provide customized output_options to be used.
If unchecked, rmarkdown will use the default output format:
Add a semi-colon delimited list of JavaScript, CSS, or library dependencies if needed.
Report Thumbnail
Choose to auto-generate a default thumbnail if desired. You can later edit the thumbnail or attach a custom image. See Manage Views.
Shared Scripts
Once you save an R report, its associated script becomes available to execute using source(“<Script Name>.R”) in future scripts.
Check the box next to the appropriate script to make it available for execution in this script.
Study Options
Participant Chart: A participant chart shows measures for only one participant at a time. Select the participant chart checkbox if you would like this chart to be available for review participant-by-participant.
Automatically cache this report for faster reloading: Check to enable.
Click Save to save settings, or Save As to save without disturbing the original saved report.
Example
Regardless of where you have accessed the R report builder, you can create a first R report which is data independent. This sample was adapted from the R help files.
Paste the following into the Source tab of the R report builder.
Click the Report tab to run the source and see your results, in this case the coin flip outcomes.
Add or Suppress Console Output
The options covered below can be included directly in your R report. There are also options related to console output in the scripting configuration for your R engine.
Echo to Console
By default, most R commands do not generate output to the console as part of your script. To enable output to console, use the following line at the start of your scripts:
options(echo=TRUE);
Note that when the results of functions are assigned, they are also not printed to the console. To see the output of a function, assign the output to a variable, then just call the variable. For further details, please see the FAQs for LabKey R Reports.
Suppress Console Output
To suppress output to the console, hiding it from users viewing the script, first remove the echo statement shown above. You can also include sink to redirect any outputs to 'nowhere' for all or part of your script.To suppress output, on Linux/Mac/Unix, use:
sink("/dev/null")
On Windows use:
sink("NUL")
When you want to restart output to the console within the script, use sink again with no argument:
Saved R reports may be accessed from the source data grid or from the Data Views web part. This topic describes how to manage saved R reports and how they can be shared with other users (who already have access to the underlying data).
Once saved, reports are generated by re-running their associated scripts on live data. This ensures users always have the most current views, but it also requires computational resources each time the view is opened. If your script is computationally intensive, you can set it to run in the background so that it does not overwhelm your server when selected for viewing. Learn more in this topic: R Report Builder.
Edit an R Report Script
Open your saved R report by clicking the name in the data views web part or by selecting it from the (Charts and Reports) menu above the data grid on which it is based. This opens the R report builder interface on the Data tab. Select the Source tab to edit the script and manage sharing and other options. Click Save when finished.
Share an R Report
Saved R Reports can be kept private to the author, or shared with other users, either with all users of the folder, or individually with specific users. Under Options in the R report builder, use the Make this report available to all users checkbox to control how the report is shared.
If the box is checked, the report will be available to any users with "Read" access (or higher) in the folder. This access level is called "public" though that does not mean shared with the general public (unless they otherwise have "Read" access).
If the box is unchecked, the report is "private" to the creator, but can still be explicitly shared with other individual users who have access to the folder.
An otherwise "private" report that has been shared with individual users or groups has the access level "custom".
When sharing a report, you are indicating that you trust the recipient(s), and your recipients confirm that they trust you when they accept it. Sharing of R reports is audited and can be tracked in the "Study events" audit log.Note that if a report is "public", i.e was made available to all users, you can still use this mechanism to email a copy of it to a trusted individual, but that will not change the access level of the report overall.
Open an R Report, from the Data Views web part, and click (Share Report).
Enter the Recipients, one per line.
The default Message Subject and Message Body are shown. Both can be customized as needed.
The Message Link is shown; you can click Preview Link to see what the recipient will see.
Click Submit to share the report. You will be taken to the permissions page.
On the Report and View Permissions page, you can see which groups and users already had access to the report.
Note that you will not see the individuals you are sharing the report with unless the access level of it was "custom" or "private" prior to sharing it now.
Click Save.
Recipients will receive a notification with a link to the report, so that they may view it. If the recipient has the proper permissions, they will also be able to edit and save their own copy of the report. If the author makes the source tab visible, recipients of a shared report will be able to see the source as well as the report contents. Note that if the recipient has a different set of permissions, they may see a different set of data. Modifications that the original report owner makes to the report will be reflected in the link as viewed by the recipient.When an R report was private but has been shared, the data browser will show access as "custom". Click custom to open the Report Permissions page, where you can see the list of groups and users with whom the report was shared.Learn more about report permissions in this topic: Configure Permissions for Reports & Views
Delete an R Report
You can delete a saved report by first clicking the pencil icon at the top of the Data Views web part, then click the pencil to the left of the report name. In the popup window, click Delete. You can also multi-select R reports for deletion on the Manage Views page.Note that deleting a report eliminates its associated script from the "Shared Scripts" list in the R report interface. Make sure that you don’t delete a script that is called (sourced) by other scripts you need.
LabKey Server automatically reads your chosen dataset into a data frame called
labkey.data using Input Substitution.A data frame can be visualized as a list with unique row names and columns of consistent lengths. Column names are converted to all lower case, spaces or slashes are replaced with underscores, and some special characters are replaced with words (i.e. "CD4+" becomes "cd4_plus_"). You can see the column names for the built in labkey.data frame by calling:
options(echo=TRUE); names(labkey.data);
Just like any other data.frame, data in a column of labkey.data can be referenced by the column's name, converted to all lowercase and preceded by a $:
labkey.data$<column name>
For example, labkey.data$pulse; provides all the data in the Pulse column. Learn more about column references below.Note that the examples in this section frequently include column names. If you are using your own data or a different version of LabKey example data, you may need to retrieve column names and edit the code examples given.
Use Pre-existing R Scripts
To use a pre-existing R script with LabKey data, try the following procedure:
Open the dataset of interest ("Physical Exam" for example).
Select > Create R Report.
Paste the script into the Source tab.
Identify the LabKey data columns that you want to be represented by the script, and load those columns into vectors. The following loads the Systolic Blood Pressure and Diastolic Blood Pressure columns into the vectors x and y:
x <- labkey.data$diastolicbp; y <- labkey.data$systolicbp;
Once you have loaded your data, you can perform statistical analyses using the functions/algorithms in R and its associated packages. For example, calculate the mean Pulse for all participants.
options(echo=TRUE); names(labkey.data); labkey.data$pulse; a <- mean(labkey.data$pulse, na.rm= TRUE); a;
Find Means for Each Participant
The following simple script finds the average values of a variety of physiological measurements for each study participant.
# Get means for each participant over multiple visits;
We use na.rm as an argument to aggregate in order to calculate means even when some values in a column are NA.
Create Functions in R
This script shows an example of how functions can be created and called in LabKey R scripts. Before you can run this script, the Cairo package must be installed on your server. See Install and Set Up R for instructions.Note that the second line of this script creates a "data" copy of the input file, but removes all participant records that contain an NA entry. NA entries are common in study datasets and can complicate display results.
filter <- function(value) { sub <- subset(labkey.data, labkey.data$participantid == value); #print("the number of rows for participant id: ") #print(value) #print("is : ") #print(sub) chart(sub) }
names(labkey.data); Cairo(file="${imgout:a}", type="png"); layout(matrix(c(1:4), 2, 2, byrow=TRUE)); strand1 <- labkey.data[,1]; for (i in strand1) { #print(i) value <- i filter(value) }; dev.off();
Access Data in Another Dataset (Select Rows)
You can use the Rlabkey library's selectRows to specify the data to load into an R data frame, including labkey.data, or a frame named something else you choose.For example, if you use the following, you will load some example fictional data from our public demonstration site that will work with the above examples.
Include colNameOpt="rname" to have the selectRows call provide "R-friendly" column names. This converts column names to lower case and replaces spaces or slashes with underscores. Note that this may be different from the built in column name transformations in the built in labkey.data frame. The built in frame also substitutes words for some special characters, i.e. "CD4+" becomes "cd4_plus_", so during report development you'll want to check using names(labkey.data); to be sure your report references the expected names.Learn more in the Rlabkey Documentation.
Select Specific Columns
Use the colSelect option with to specify the set of columns you want to add to your dataframe. Make sure there are no spaces between the commas and column names.In this example, we load some fictional example data, selecting only a few columns of interest.
If you load the above example, and then execute: labkey.data$language; you will see all the data in the "Language" column.
Remember that in an R data frame, columns are referenced in all lowercase, regardless of casing in LabKey Server. For consistency in your selectRows call, you can also define the colSelect list in all lowercase, but it is not required.
If "Language" were a lookup column referencing a list of Languages with accompanying translators, this would return a series of integers or whatever the key of the "Language" column is.You could then access a column that is not the primary key in the lookup target, typically a more human-readable display values. Using this example, if the "Translator" list included "LanguageName, TranslatorName and TranslatorPhone" columns, you could use syntax like this in your selectRows,
You can now retrieve human-readable values from within the "Language" list by converting everything to lowercase and substituting an underscore for the slash. Executing labkey.data$language_languagename; will return the list of language names.
Access URL Parameters and Data Filters
While you are developing your report, you can acquire any URL parameters as well as any filters applied on the Data tab by using labkey.url.params.For example, if you filter the "systolicBP" column to values over 100, then use:
print(labkey.url.params}
...your report will include:
$`Dataset.systolicBP~gt` [1] "100"
Write Result File to File Repository
The following report, when run, creates a result file in the server's file repository. Note that fileSystemPath is an absolute file path. To get the absolute path, see Using the Files Repository.
The scripts on this page take the analysis techniques introduced in R Reports: Access LabKey Data one step further, still using the Physical Exam sample dataset. This page covers a few more strategies for finding means, then shows how to graph these results and display least-squares regression lines.
Find Mean Values for Each Participant
Finding the mean value for physiological measurements for each participant across all visits can be done in various ways. Here, we cover three alternative methods.For all methods, we use "na.rm=TRUE" as an argument to aggregate in order to ignore null values when we calculate means.
Description
Code
Aggregate each physiological measurement for each participant across all visits; produces an aggregated list with two columns for participantid.
Aggregate only the pulse column and display two columns: one listing participantIDs and the other listing mean values of the pulse column for each participant
Next we use R to create plots of some other physiological measurements included in our sample data.
All scripts in this section use the Cairo package. To convert these scripts to use the png() function instead, eliminate the call "library(Cairo)", change the function name "Cairo" to "png," change the "file" argument to "filename," and eliminate the "type="png"" argument entirely.
Scatter Plot of All Diastolic vs All Systolic Blood Pressures
This script plots diastolic vs. systolic blood pressures without regard for participantIDs. It specifies the "ylim" parameter for plot() to ensure that the axes used for this graph match the next graph's axes, easing interpretation.
The generated plot, where the identity of participants is ignored, might look like this:
Scatter Plot of Mean Diastolic vs Mean Systolic Blood Pressure for Each Participant
This script plots the mean diastolic and systolic blood pressure readings for each participant across all visits. To do this, we use "data_means," the mean value for each physiological measurement we calculated earlier on a participant-by-participant basis.
This time, the plotted regression line for diastolic vs. systolic pressures shows a non-zero slope. Looking at our data on a participant-by-participant basis provides insights that might be obscured when looking at all measurements in aggregate.
Create Multiple Plots
There are two ways to get multiple images to appear in the report produced by a single script.
Single Plot Per Report Section
The first and simplest method of putting multiple plots in the same report places separate graphs in separate sections of your report. Use separate pairs of device on/off calls (e.g., png() and dev.off()) for each plot you want to create. You have to make sure that the {imgout:} parameters are unique. Here's a simple example:
There are various ways to place multiple plots in a single section of a report. Two examples are given here, the first using par() and the second using layout().Example: Four Plots in a Single Section: Using par()This script demonstrates how to put multiple plots on one figure to create a regression panel layout. It uses standard R libraries for the arrangement of plots, and Cairo for creation of the plot image itself. It creates a single graphics file but partitions the ‘surface’ of the image into multiple sections using the mfrow and mfcol arguments to par().
Example: Three Plots in a Single Section: Using layout()This script uses the standard R libraries to display multiple plots in the same section of a report. It uses the layout() command to arrange multiple plots on a single graphics surface that is displayed in one section of the script's report.The first plot shows blood pressure and weight progressing over time for all participants. The lower scatter plots graph blood pressure (diastolic and systolic) against weight.
The "lattice" R package provides presentation-quality, multi-plot graphics. This page supplies a simple script to demonstrate the use of Lattice graphics in the LabKey R environment.Before you can use the Lattice package, it must be installed on your server.
You will load the lattice package at the start of every script that uses it:
library("lattice");
Display a Volcano
The Lattice Documentation on CRAN provides a Volcano script to demonstrate the power of Lattice. The script below has been modified to work on LabKey R:
You can use the Participant Chart checkbox in the R Report Builder to create charts that display your R report results on a participant-by-participant basis.
Click the Report tab to view the scatter plot data for all participants.
Return to the Source tab.
Scroll down and click the triangle to open the Study Options section.
Check Participant Chart.
Click Save.
Name your report "Participant Systolic" or another name you choose.
The participant chart option subsets the data that is handed to an R script by filtering on a participant ID. You can later step through per participant charts using this option. The labkey.data dataframe may contain one, or more rows of data depending on the content of the dataset you are working with. Next, reopen the R report:
Return to the data grid of the "PhysicalExam" dataset.
Select (Charts/Reports) > Participant Systolic (or the name you gave your report).
Click Previous Participant.
You will see Next Participant and Previous Participant links that let you step through charts for each participant:
Advanced Example: Create Participant Charts Using Lattice
You can create a panel of charts for participants using the lattice package. If you select the participant chart option on the source tab, you will be able to see each participant's panel individually when you select the report from your data grid.The following script produces lattice graphs for each participant showing systolic blood pressure over time:
library(lattice); png(filename="${imgout:a}", width=900); plot.new(); xyplot(systolicbp ~ date| participantid, data=labkey.data, type="a", scales=list(draw=FALSE)); update(trellis.last.object(), strip = strip.custom(strip.names = FALSE, strip.levels = TRUE), main = "Systolic over time grouped by participant", ylab="Systolic BP", xlab=""); dev.off();
The following script produces lattice graphics for each participant showing systolic and diastolic blood pressure over time (points instead of lines):
xyplot(systolicbp + diastolicbp ~ date | participantid, data=labkey.data, type="p", scales=list(draw=FALSE)); update(trellis.last.object(), strip = strip.custom(strip.names = FALSE, strip.levels = TRUE), main = "Systolic & Diastolic over time grouped by participant", ylab="Systolic/Diastolic BP", xlab=""); dev.off();
After you save these two R reports with descriptive names, you can go back and review individual graphs participant-by-participant. Use the (Reports) menu available on your data grid.
The knitr visualization package can be used with R in either HTML or Markdown pages to create dynamic reports. This topic will help you get started with some examples of how to interweave R and knitr.
If you are using rmarkdown v2 and check the box to "Use advanced rmarkdown output_options (pandoc only)", you can enter a list of param=value pairs in the box provided. Enter the bolded portion of the following example, which will be enclosed in an "output_options=list()" call.
Supported options include those in the "html_document" output format. Learn more in the rmarkdown documentation here .Sample param=value options you can include:
css: Specify a custom stylesheet to use.
fig_width and fig_height: Control the size of figures included.
fig_caption: Control whether figures are captioned.
Note that pandoc is only supported for rmarkdown v2, and some formats supported by pandoc are not supported here.If you notice report issues, such as graphs showing as small thumbnails, or HTML not rendering as expected in R reports, you may need to upgrade your server's version of pandoc.
R/knitr Scripts in Modules
R script knitr reports are also available as custom module reports. The script file must have either a .rhtml or .rmd extension, for HTML or markdown documents, respectively. For a file-based module, place the .rhtml/.rmd file in the same location as .r files, as shown below. For module details, see Map of Module Files.
To fully utilize the report designer (called the "R Report Builder" in the LabKey user interface), you can declare JavaScript or CSS dependencies for knitr reports. This ensures that the dependencies are downloaded before R scripts are run on the "reports" tab in the designer. If these dependencies are not specified then any JavaScript in the knitr report may not run correctly in the context of the script designer. Note that reports that are run in the context of the Reports web part will still render correctly without needing to explicitly define dependencies.Reports can either be created via the LabKey Server UI in the report designer directly or included as files in a module. Reports created in the UI are editable via the Source tab of the designer. Open Knitr Options to see a text box where a semi-colon delimited list of dependencies can be entered. Dependencies can be external (via HTTP) or local references relative to the labkeyWebapp path on the server. In addition, the name of a client library may be used. If the reference does not have a .js or .css extension then it will be assumed to be a client library (somelibrary.lib.xml). The .lib.xml extension is not required. Like local references, the path to the client library is relative to the labkeyWebapp path.File based reports in a module cannot be edited in the designer although the "source" tab will display them. However you can still add a dependencies list via the report's metadata file. Dependencies can be added to these reports by including a <dependencies> section underneath the <R> element. A sample metadata file:
<?xml version="1.0" encoding="UTF-8"?> <ReportDescriptor xmlns="http://labkey.org/query/xml"> <label>My Knitr Report</label> <description>Relies on dependencies to display in the designer correctly.</description> <reportType> <R> <dependencies> <dependency path="http://external.com/jquery/jquery-1.9.0.min.js"/> <dependency path="knitr/local.js"/> <dependency path="knitr/local.css"/> </dependencies> </R> </reportType> </ReportDescriptor>
The metadata file must be named <reportname>.report.xml and be placed alongside the report of the same name under (modulename/resources/reports/schemas/...).
Copy and paste the knitr code below into the Source tab of the R Report Builder.
Scroll down to the Knitr Options node, open the node, and select HTML.
Click the Report tab to see the knitr report.
<table> <tr> <td align='center'> <h2>Scatter Plot: Blood Pressure</h2> <!--begin.rcode echo=FALSE, warning=FALSE library(ggplot2); opts_chunk$set(fig.width=10, fig.height=6) end.rcode--> <!--begin.rcode blood-pressure-scatter, warning=FALSE, message=FALSE, echo=FALSE, fig.align='center' qplot(labkey.data$diastolicbp, labkey.data$systolicbp, main="Diastolic vs. Systolic Pressures: All Visits", ylab="Systolic (mm Hg)", xlab="Diastolic (mm Hg)", ylim =c(60, 200), xlim=c(60,120), color=labkey.data$temp); end.rcode--> </td> <td align='center'> <h2>Scatter Plot: Body Temp vs. Body Weight</h2> <!--begin.rcode temp-weight-scatter, warning=FALSE, message=FALSE, echo=FALSE, fig.align='center' qplot(labkey.data$temp, labkey.data$weight, main="Body Temp vs. Body Weight: All Visits", xlab="Body Temp (C)", ylab="Body Weight (kg)", xlim=c(35,40), color=labkey.data$height); end.rcode--> </td> </tr> </table>
The rendered knitr report:
Markdown v2
Administrators can enable Markdown v2 when enlisting an R engine through the Views and Scripting Configuration page. When enabled, Markdown v2 will be used when rendering knitr R reports. If not enabled, Markdown v1 is used to execute the reports.Independent installation is required of the following:
any other package dependencies on the same server as the R engine
This will then enable using the Rmarkdown v2 syntax for R reports. The system does not currently perform any verification of the user's setup. If the configuration is enabled when enlisting the R engine, but the packages are not properly setup, the intended report rendering will fail.Syntax differences are noted here: http://rmarkdown.rstudio.com/authoring_migrating_from_v1.html
Markdown v1 Example
# Scatter Plot: Blood Pressure # The chart below shows data from all participants
```{r setup, echo=FALSE} # set global chunk options: images will be 7x5 inches opts_chunk$set(fig.width=7, fig.height=5) ```
Premium Resource AvailableSubscribers to premium editions of LabKey Server can learn how to incorporate a plotly graph with the example code in this topic:
Proxy Servlets: Another way to use plotly with LabKey data.
Input/Output Substitutions Reference
An R script uses input substitution parameters to generate the names of input files and to import data from a chosen data grid. It then uses output substitution parameters to either directly place image/data files in your report or to include download links to these files. Substitutions take the form of: ${param} where 'param' is the substitution.
You can find the substitution syntax directly in the R Report Builder on the Help tab.
Input and Output Substitution Parameters
Valid Substitutions:
input_data: <name>
The input datset, a tab-delimited table. LabKey Server automatically reads your input dataset (a tab-delimited table) into the data frame called labkey.data. If you desire tighter control over the method of data upload, you can perform the data table upload yourself. The 'input data:' prefix indicates that the data file for the grid and the <name> substitution can be set to any non-empty value:
An image output file (such as jpg, png, etc.) that will be displayed as a Section of a View on LabKey Server. The 'imgout:' prefix indicates that the output file is an image and the <name> substitution identifies the unique image produced after you call dev.off(). The following script displays a .png image in a View:
A PDF output file that can be downloaded from LabKey Server. The 'pdfout:' prefix indicates that he expected output is a pdf file. The <name> substitution identifies the unique file produced after you call dev.off().
A postscript output file that can be downloaded from LabKey Server. The 'psout:' prefix indicates that the expected output is a postscript file. The <name> substitution identifies the unique file produced after you call dev.off().
A file output that can be downloaded from LabKey Server, and may be of any file type. For example, use fileout in the place of tsvout to allow users to download a TSV instead of seeing it within the page:
A text file that is displayed on LabKey Server as a section within a View. The output is different from the txtout: replacement in that no html escaping is done. This is useful when you have a report that produces html output. No downloadable file is created:
txt <- paste("<i>Click on the link to visit LabKey:</i> <a target='blank' href='https://www.labkey.org'>LabKey</a>") # ${htmlout:output} write(txt, file="output")
svgout: <name>
An svg file that is displayed on LabKey Server as a section within a View. htmlout can be used to render svg outputs as well, however, using svgout will generate a more appropriate thumbnail image for the report. No downloadable file is created:
Each R script contains implicit variables that are inserted before your source script. Implicit variables are R data types and may contain information that can be used by the source script.
Implicit variables:
labkey.data
The data frame into which the input dataset is automatically read. The code to generate the data frame is:
The path portion of the current URL which omits the base context path, action and URL parameters. The path portion of the URL: http://localhost:8080/home/test/study-begin.view would be: /home/test/
The list of parameters on the current URL and in any data filters that have been applied. The parameters are represented as a list of key / value pairs.
labkey.user.email
The email address of the current user
Using Regular Expressions with Replacement Token Names
Sometimes it can be useful to have flexibility when binding token names to replacement parameters. This can be the case when a script generates file artifacts but does not know the file names in advance. Using the syntax: regex() in the place of a token name (where LabKey server controls the token name to file mapping) will result the following actions:
Any script generated files not mapped to a replacement will be evaluated against the file's name using the regex.
If a file matches the regex, it will be assigned to the replacement and rendered accordingly.
<replacement>:regex(<expression>)
The following example will find all files generated by the script with the extension : '.gct'. If any are found they will be assigned and rendered to the replacement parameter (in this case as a download link).//
#${fileout:regex(.*?(\.gct))}
Cairo or GDD Packages
You may need to use the Cairo or GDD graphics packages in the place of jpeg() and png() if your LabKey Server runs on a "headless" Unix server. You will need to make sure that the appropriate package is installed in R and loaded by your script before calling either of these functions.GDD() and Cairo() Examples. If you are using GDD or Cairo, you might use the following scripts instead:
Note that the auto-generated code will include any filters applied to the grid. For example, if you filter Physical Exam for records where temperature is greater than 37, then the auto-generated code will include a colFilter property as below:
library(Rlabkey)
# Select rows into a data frame called 'labkey.data'
This page aims to answer common questions about configuring and using the LabKey Server interface for creating R Reports. Remember, an administrator must install and configure R on LabKey Server before users can create and run R scripts on live datasets.Topics:
library(), help() and data() don’t work
plot() doesn’t work
jpeg() and png() don’t work
Does my report reflect live, updated data?
Output is not printed when I source() a file or use a function
Scripts pasted from documentation don't work in the LabKey R Script Builder
LabKey Server becomes very, very slow when scripts execute
Does R create security risks?
Any good sources for advice on R scripting?
1. library(), help() and data() don’t work
LabKey Server runs R scripts in batch mode. Thus, on Windows machines it does not display the pop-up windows you would ordinarily see in R’s interpreted/interactive mode. Some functions that produce pop-ups (e.g., library()) have alternatives that output to the console. Some functions (e.g., help() and some forms of data()) do not.Windows Workaround #1: Use alternatives that output to the consolelibrary(): The library() command has a console-output alternative. To see which packages your administrator has made available, use the following:
installed.packages()[,0]
Windows Workaround #2: Call the function from a native R windowhelp(): It’s usually easy to keep a separate, native R session open and call help() from there. This works better for some functions than others. Note that you must install and load packages before asking for help() with them. You can also use the web-based documentation available on CRAN.data(): You can also call data() from a separate, native R session for some purposes. Calling data() from such a session can tell you which datasets are available on any packages you’ve installed and loaded in that instance of R, but not your LabKey installation.
2. plot() doesn’t work
Did you open a graphics device before calling plot()?LabKey Server executes R scripts in batch mode. Thus, LabKey R never automatically opens an appropriate graphics device for output, as would R when running in interpreted/interactive mode. You’ll need to open the appropriate device yourself. For onscreen output that becomes part of a report, use jpeg() or png() (or their alternatives, Cairo(), GDD() and bitmap()). In order to output a graphic as a separate file, use pdf() or postscript().Did you call dev.off() after plotting?You need to call dev.off() when you’re done plotting to make sure the plot object gets printed to the open device.
3. jpeg() and png() don’t work
R is likely running in a headless Unix server. On a headless Unix server, R does not have access to the appropriate X11 drivers for the jpeg() and png() functions. Your admin can install a display buffer on your server to avoid this problem. Otherwise, in each script you will need to load the appropriate package to create these file formats via other functions (e.g., GDD or Cairo).
4. Does my report reflect live, updated data?
Yes. In general, LabKey always re-runs your saved script before displaying its associated report. Your script operates on live, updated data, so its plots and tables reflect fresh data.In study folders, you can set a flag for any script that prevents the script from being re-run unless changes have occurred. This flag can save time when scripts are time-intensive or datasets are large making processing slow. When this flag is set, LabKey will only re-run the R script if:
The flag is cleared OR
The dataset associated with the script has changed OR
Any of the attributes associated with the script are changed (script source, options etc.)
To set the flag, check the "Automatically cache this report for faster reloading" checkbox under "Study Options" on the Source tab of the R report builder.
5. Output is not printed when I source() a file or use a function
When you use… functions interactively at the command line, the result is automatically printed...In source() or inside your own functions you will need an explicit print() statement.
When a command is executed as part of a file that is sourced, the command is evaluated but its results are not ordinarily printed. For example, if you call source(scriptname.R) and scriptname.R calls installed.packages()[,0]
, the installed.packages()[,0]
command is evaluated, but its results are not ordinarily printed. The same thing would happen if you called installed.packages()[,0]
from inside a function you define in your R script.You can force sourced scripts to print the results of the functions they call. The R FAQ explains:
If you type `1+1' or `summary(glm(y~x+z, family=binomial))' at the command line the returned value is automatically printed (unless it is invisible()). In other circumstances, such as in a source()'ed file or inside a function, it isn't printed unless you specifically print it.
To print the value 1+1, use
print(1+1);
or, instead, use
source("1plus1.R", echo=TRUE);
where "1plus1.R" is a shared, saved script that includes the line "1+1".
6. Scripts pasted from documentation don't work in the LabKey R report builder
If you receive an error like this:
Error: syntax error, unexpected SYMBOL, expecting 'n' or ';' in "library(Cairo) labkey.data" Execution halted
please check your script for missing line breaks. Line breaks are known to be unpredictably eliminated during cut/paste into the script builder. This issue can be eliminated by ensuring that all scripts have a ";" at the end of each line.
7. LabKey Server becomes very, very slow when scripts execute
You are probably running long, computationally intensive scripts. To avoid a slowdown, run your script in the background via the LabKey pipeline. See R Report Builder for details on how to execute scripts via the pipeline.
8. Does R Create Security Risks?
Allowing the use of R scripts/reports on a server can be a security risk. A developer could write a script that could read or write any file stored in any SiteRoot, fileroot or pipeline root despite the LabKey security settings for that file.A user must have at least the Author role as well as either the Platform Developer or Trusted Analyst role to write a R script or report to be used on the server.R should not be used on a "shared server", that is, a server where users with admin/developer privileges in one project do not have permissions on other projects. Running R on the server could pose a security threat if the user attempts to access the server command line directly. The main way to execute a system command in R is via the 'system(<system call>)' method that is part of the R core package. The threat is due to the permission level of a script being run by the server possibly giving unwanted elevated permissions to the user.Administrators should investigate sandboxing their R configuration as a software management strategy to reduce these risks. Note that the LabKey Server will not enforce or confirm such sandboxing, but offers the admin a way of announcing that an R configuration is safe.
Lectures and labs cover the range - from introductory R to advanced genomic analysis
10. Graphics File Formats
If you don’t know which graphics file format to use for your plots, this link can help you narrow down your options.
.png and .gif
Graphics shared over the web do best in png when they contain regions of monotones with hard edges (e.g., typical line graphs). The .gif format also works well in such scenarios, but it is not supported in the default R installation because of patent issues. The GDD package allows you to create gifs in R.
.jpeg
Pictures with gradually varying tones (e.g., photographs) are successfully packaged in the jpeg format for use on the web.
.pdf and .ps or .eps
Use pdf or postscript when you aim to output a graph that can be accessed in isolation from your R report.
Premium RStudio Integration
Premium Feature — Available in the Professional and Enterprise Editions of LabKey Server. Learn more or contact LabKey.
The RStudio module lets you design and save reports using RStudio from Posit, instead of LabKey Server's built in R report designer.There are two options for using RStudio with LabKey.
Once either RStudio or RStudio Workbench is configured and enabled using one of the above methods, you can use it to edit R reports. You can also directly export data into a named variable. Learn more in these topics:
Premium Feature — This feature supports RStudio, which is part of the Professional and Enterprise Editions of LabKey Server. Learn more or contact LabKey.
This topic explains how administrators can set up communication between LabKey Server and an RStudio instance packaged inside a Docker image. As part of the configuration, we recommend that you allow Docker, not LabKey Server, to manage the RStudio data using Docker Volumes. This avoids a number of issues with file sharing, permissions, path mapping, and security, making setup much easier.
Depending on your platform and server mode, you have several connection options:
Connect using UNIX SOCKET
On Mac and Linux, you can connect directly via the docker socket, usually at
unix:///var/run/docker.sock
Connect using TCP - Developer mode
If you are developing or testing locally, you can skip setting up TLS if both of the following are true:
Use port 2375 instead of 2376
Run with devmode=true
Otherwise a secure connection is required, as described in the next section, whether the docker server is local or remote.
Connect using TCP and TLS - Production mode
In production you must set up TLS if you are connecting over the network, and you may also want to set it up for certain kinds of development. Set up TLS following these steps:
After certificates are set up, find the values of the following environment variables. You will use these values in the Configure Docker step below. (On Linux run 'env | grep DOCKER'.)
DOCKER_CERT_PATH
DOCKER_HOST
On an Ubuntu system modify the "/etc/default/docker" file to include the following:
Clone the docker-rstudio repository from github: https://github.com/LabKey/docker-rstudio (This is not a module and should not be cloned into your LabKey server enlistment.)
cd to the rstudio-base directory:
cd docker-rstudio/images/labkey/rstudio-base
Build the image from the source files.
On Linux:
./make
On Windows
make.bat
Test by running the following:
docker run labkey/rstudio-base
You may need to provide a password (generate a Session Key to use as a password) to perform this test, but note that this test is simulating how RStudio will be run from LabKey and you do not need to leave this container running or use a persistent password or API key.
docker run -e PASSWORD=<Your_Session_Key> -p 8787:8787 labkey/rstudio-base
Check console output for errors.
Ctrl-C to get to return to the command prompt once you see the 'done' message.
Stop this test container by running:
docker stop labkey/rstudio-base
Enable Session Keys
Your server must be configured to accept session keys in order for token authentication to RStudio to work. One time access codes will be generated for you.
You do not need to generate any session keys yourself to use RStudio.
Check Base Server URL
Go to > Site > Admin Console.
Under Configuration, click Site Settings.
The Base Server URL cannot be localhost. For a local development machine, you can substitute your machine's actual IP address (obtain via ipconfig) in this entry.
Note that you may have to log out of "localhost" and log in to the actual IP address to use RStudio from a local server.
If you have successfully created the Docker image, it will be listed under Docker Volumes.
When configuring Docker for RStudio, you can skip the prerequisite check for user home directories.
The configuration page will show a list of existing volumes and corresponding user emails. The email field is blank if the volume doesn't correspond to an existing user.
Configure RStudio
Go to > Site > Admin Console.
Under Premium Features click RStudio Settings.
Select Use Docker Based RStudio.
Configure the fields as appropriate for your environment, then click Save.
Image Configuration
Docker Image Name: Enter the Docker image name to use for RStudio. Default is 'labkey/rstudio', which includes Rlabkey.
Port: Enter port the RStudio Server will listen on inside Docker. Default is 8787.
Mount Volume: Enter the volume inside the Docker container to mount as the RStudio user's home directory. Default is '/home/rstudio'.
Host library directory (read-only): Optional read-only mount point at /usr/lib/R/library.
Additional mount: host/container directories (read-only): Optional read-only mount points.
User for Containers: Optional user to use for running containers inside docker.
AppArmor Profile: Provide if needed.
Local IP address for LabKey Server: Set to the IP address of the LabKey Server from within your docker container. This may be left blank if the normal DNS lookup works.
Rstudio User Configuration:
User: User name for rstudio user. Should be 'rstudio' unless container is in rootless mode in which case it should be 'root'.
User ID: UserId to use for rstudio user (probably same as the userid of the tomcat service).
Group: Login group for the Rstudio user
Group ID: Login group ID.
Additional Group: Additional security group for rstudio user
Additional Group ID: Additional security group ID
Rootless Mode: Check the box to run the docker container in 'rootless mode'. When this is checked, the user field above should be 'root'.
Resource Configuration: Customize defaults if needed.
Max User Count: Default is 10.
Stop Container Timeouts (in hours)": Default is 1.0. Click to Stop Now.
Delete Container Timeouts (in hours)": Default is 120.0. Click to Delete Now.
Click Save when finished. A button to Restore Defaults is also available.
Enable RStudio in Folders
The modules docker and rstudio must be enabled in each folder where you want RStudio to be available. Select > Folder > Management, select the Folder Type tab and make sure both modules are checked in the right column. Click Update Folder to save any changes.Once RStudio is enabled, you can use the instructions in these topics to edit reports and export data:
Premium Feature — Docker supports RStudio, which is part of the Professional and Enterprise Editions of LabKey Server. Learn more or contact LabKey for details.
This topic outlines an example TLS and certificate configuration used to set up Docker for use with RStudio in LabKey Server. The main instructions for configuring Docker and RStudio are in this topic: Connect to RStudio
On a development machine, you do not have to set up TLS if both of the following are true:
Use port 2375 instead of 2376
Run with devmode=true
Otherwise a secure connection is required, whether the docker server is local or remote.
Installation Instructions for Docker Daemon
First, identify where the server will run the Docker Daemon on the Test Environment.The Docker documentation includes the following recommendation: "...if you run Docker on a server, it is recommended to run exclusively Docker in the server, and move all other services within containers controlled by Docker. Of course, it is fine to keep your favorite admin tools (probably at least an SSH server), as well as existing monitoring/supervision processes (e.g., NRPE, collectd, etc)."The options available include:
Create a New Instance in the Test Environment for the Docker Daemon. This option is preferred in an ideal world, as it is the most secure and follows all Docker best practices. If a runaway RStudio process were to overwhelm the instance's resources it would not affect Rserve and the Web Server processes and thus the other applications would continue to function properly.
Install and Run the Docker Daemon on the Rserve instance. This allows quick installation and configuration of Docker in the Test Environment.
Install and Run the Docker Daemon on the Web Server.
For the Test Environment, option #2 is recommended as this allows us to test the RStudio features using a configuration similar to what will be used in Production.
If during the testing, we begin to see resource contention between RStudio and Rserve processes, we can change the Docker Daemon configuration to limit resources used by RStudio or request a new instance to run Docker.
If a runaway RStudio process overwhelms the instance's resources it will not affect the Web Server processes and thus the other applications will continue to function properly.
Install the Docker Daemon
Install the Docker Daemon on the Rserve instance by running the following commands:
IMPORTANT: Do not add any users to the docker group (members of the docker group can access dockerd Linux socket).
Create TLS Certificates
The TLS certificates are used by the LabKey Server to authenticate to the Docker Daemon process.Create the directory that will hold the CA certificate/key and the Client certificate/key. You can use a different directory if you want than the one shown below. This is the value of "DOCKER_CERT_PATH":
sudo su - mkdir -p /labkey/apps/ssl chmod 700 /labkey/apps/ssl
Create the Certificate Authority private key and certificate
Create the CA key CA certificate. Configure the certificate to expire in 10 years.
Create the TLS certificates to be used by the Docker Daemon
Create the directory from which the docker process will read the TLS certificate and key files.
mkdir /etc/docker/ssl chmod 700 /etc/docker/ssl/
Copy over the CA certificate.
cp /labkey/apps/ssl/ca.pem /etc/docker/ssl
Create the openssl.cnf configuration file to be used by the Docker Daemon. Note: Values for Test and Production are shown separated by "|" pipes. Keep only the option needed.
Note: Values for Test and Production are shown separated by "|" pipes. Keep only the option needed.Change the Docker Daemon to use your preferred configuration. The changes are:
Daemon will listen Linux socket at "/var/run/docker.sock" and "tcp://10.0.1.204 | 10.10.1.74:2376"
Use TLS certificates on the TCP socket for encryption and authentication
Turn off inter-container communication
Set default limits on container usage and enable userland-proxy
These options were set creating the daemon.json configuration file at:
Now that the profile is loaded, we can force the container to use the profile by specifying it at run time, for example:
docker run --security-opt "apparmor=docker-labkey-myserver" -i -t ubuntu /bin/bash
When starting a new Docker container manually, you should always use the "docker-labkey-myserver" profile. This is done by specifying the following option whenever a container is started:
--security-opt "apparmor=docker-labkey-myserver"
Install New Firewall Rules
Install file containing new firewall rules to block certain connections from running containers
Note: Values for Test and Production are shown separated by "|" pipes. Keep only the option needed.
mkdir /etc/iptables.d vi /etc/iptables.d/docker-containers [added]
Changes to RServe Instance to Support RStudio Usage
In order to support file sharing between the RStudio process and the LabKey Server we will need to:
Create new user accounts on Rserve instance
Create new groups on Rserve instance
Modify permission on the /share/users directory
Add new groups and user accounts Rserve instance
Create the new group named "labkey-docker" this group is created to facilate the file sharing.
sudo groupadd -g 6000 labkey-docker
Create the new RStudio user. This will be the OS user which runs the RStudio session
sudo useradd -u 6005 -m -G labkey-docker rstudio
Create the directory which will hold the Home Directory Fileroots used by each server
We will use acls to ensure that newly created files/directories have the correct permissions. The ACLs only need to be set on the Rserve operating system. They do not need to specified or changed on the container.These ACLs should only be specified on the LabKey Server Home Directory FileRoot. Never set these ACLs on any other directory on the Rserve. Sharing files between the docker host and container should only be done in the LabKey Server Home Directory FileRoot.Install required package:
sudo apt-get install acl
Create the home dir fileroot and set the correct permissions using the newly created accounts. These instructions assume:
Tomcat server is being run by the user "myserver"
LabKey Server Home Directory FileRoot is located at "/share/users"
Premium Feature — This feature supports RStudio, which is part of the Professional and Enterprise Editions of LabKey Server. Learn more or contact LabKey for details.
RStudio Workbench (previously RStudio Server Pro) supports running multiple RStudio sessions per user, and multiple versions of R simultaneously.
Once RStudio Workbench is correctly configured, the RStudio Workbench Admin should provide you with:
The absolute URL of RStudio Workbench
The Security Token customized to authenticate LabKey. Default is 'X-RStudio-Username'
You may also wish to confirm that RStudio Workbench was restarted after setting the above.
Version Compatibility
LabKey Server version 23.3 has been confirmed to work with RStudio Workbench version 2022.07.1.
Configure RStudio Workbench in the Admin Console
Select > Site > Admin Console.
Under Configuration, click Site Settings.
Set the Base Server URL (i.e. not "localhost").
Click Save.
Click Settings
Under Premium Features, click RStudio Settings.
Select Use RStudio Workbench.
Enter the Configuration values:
RStudio Workbench URL: The absolute URL of RStudio Workbench.
Authentication: Select either Email or Other User Attribute:
Email: Provide the User Email Suffix: Used for LabKey to RStudio Workbench user mapping. The RStudio Workbench user name is derived from removing this suffix (Ex. "@labkey.com") from the user's LabKey Server email address.
Other User Attribute: For example, LDAP authentication settings can be configured to retrieve user attributes at authentication time. Provide the Attribute Name for this option.
Security Token: The HTTP header name that RStudio is configured to use to indicate the authenticated user. Default is 'X-RStudio-Username'.
Initialize Schema Viewer: Check the box to enable the viewing of the LabKey schema from within RStudio. See Advanced Initialization of RStudio for more information.
Before saving, you can use Test RStudio Workbench to confirm the connection and security token. When successful the message will read:
RStudio Workbench server connection test succeeded. Launch RStudio to verify $rStudioUserName$ is a valid RStudio user.
Proceed to launch RStudio Workbench via > Developer Links > RStudio Server. If the rstudioUserName you provided is valid, you will be logged in successfully. If not, you will see a warning message provided by RStudio Workbench.
Note that previous versions of Rstudio Server Workbench/Pro allowed LabKey to determine validity of the username. Find documentation for earlier versions in our archives: Testing RStudio Workbench/Pro Connections
Troubleshooting unsuccessful results of Test RStudio Workbench:
An error response may indicate the incorrect URL for RStudio Workbench was entered, or that URL is unreachable. Try reaching the URL in a browser window outside of LabKey.
A warning response may be due to an invalid security token.
Launch RStudio Workbench
Select > Developer Links > RStudio Server.
When you launch RStudio Workbench, you will see your home page. If you have multiple sessions, click on one to select it.Now you can use RStudio Workbench to edit R reports and export data. Learn more in these topics:
Premium Feature — This feature supports RStudio, which is part of the Professional and Enterprise Editions of LabKey Server. Learn more or contact LabKey.
This topic covers setting up and configuring RStudio Workbench (previously RStudio Server Pro) for use with LabKey Server. These instructions will not work with the RStudio Server Open Source version but only work for RStudio Workbench. RStudio Workbench runs as a separate server, and unlike RStudio, can shared by many simultaneous users. On the machine where RStudio Workbench is running, the R language must also be installed and the Rlabkey package loaded.Typically the steps on this page would be completed by a system administrator, though not necessarily the LabKey site administrator. Once the steps described here have been completed on the RStudio Workbench machine, the LabKey Server can be configured to connect to RStudio Workbench as described in Connect to RStudio Workbench. Key pieces of information from this setup must be communicated to the LabKey admin.
If you're setting this server up in AWS, you will want to set it up in a public subnet in the same VPC as the destination server. It will need a public-facing IP/EIP.
For AWS, the following ports need to be open, inbound and outbound, for this install to work in AWS. Set your security group/NACLs accordingly:
80
443
1080 (license server)
8787 (initial connection)
1024-65535 (license server)
Linux and Ubuntu Setup
If your RStudio Workbench host machine is running an OS other than Linux/Ubuntu, you can configure a VM to provide a local Linux environment in which you can run Ubuntu. If your host is Linux running Ubuntu, you can skip this section.Steps to set up Ubuntu on a non Linux host OS:
VM: Use Virtual Box to set up a local Linux environment.
Linux: Install Linux OS in that VM
Ubuntu: Download and install the latest Ubuntu (version 12.04 or higher is required) in the VM.
Configure VM network so that VM has an IP and can be accessed by the host. For example: VM > Settings > Network > Attached To > Bridged Adapter
R Language Setup (for Ubuntu)
Use the following steps to set up R for Ubuntu. For other versions of linux, follow the instructions here instead: http://docs.rstudio.com/ide/server-pro/index.html#installation1. Configure CRAN Repository. Add the following entry to your /etc/apt/sources.list file, substituting the correct name for your Ubuntu version where this example uses 'xenial' (the name for 16.x):
1. Download and install RStudio Workbench as instructed when you purchased it. For demo purposes, you can use the free 45 day trial license and download from the RStudio download page.
This function requires high ports to be world-accessible. If this command fails, check your security groups or NACLs; if this fails with a timeout, check them first.
3. Verify RStudio Workbench Installation
Create a ubuntu user:
useradd -m -s /bin/bash username passwd username
Get Ubuntu’s ip address by:
$ ifconfig
In VM browser, go to localhost:8787, or ip:8787
You should be on the RStudio Workbench login page, login using your ubuntu login
4. Verify accessing RStudio Workbench from the host machineIn the host machine’s browser, go to your VMs IP address. Depending on how you configured your license, it might look something like "http://ubuntuVmIPAddress:8787". You should see the RStudio login page and being able to log in.
Troubleshoot: did you configure VM to use Bridged Adapter?
5. Create a Route 53 DNS name for the server. You will need this for the next steps.
Configure RStudio Workbench Proxy
The RStudio Workbench administrator must enable proxy authentication on RStudio Workbench to use LabKey Server as the authenticator. The following steps will configure the proxy. See below for the details that must then be used when configuring LabKey Server to use this proxy. Note that this option is not supported for RStudio Server Open source and only works for RStudio Workbench.1. You will need to create an SSL cert:
2. Use LabKey Server as proxy authenticator for RStudio Workbench by adding the following lines to /etc/rstudio/rserver.conf, supplying your own server name:
keytool -import -keystore /usr/lib/jvm/jdk1.8.0_151/jre/lib/security/cacerts -alias <RStudio hostname, but not the FQDN> -file <whatever you named the saved cert>.crt (nb: you will be prompted for a password; ask Ops for it)
service tomcat_lk restart
7. Verify Proxy redirect
Go to the RStudio Workbench url to verify the page redirects to LabKey. Note: Once SSL is enabled, 8787 no longer works.
8. Customize Security TokenDefine a custom security token for the proxy. This will be used to indicate the authenticated user when LabKey Server connects to RStudio Workbench. For example, to define the default 'X-RStudio-Username' token:
LabKey users are mapped to RStudio users by removing the configured "email suffix". For example: LabKey user XYZ@email.com is mapped to RStudio user name XYZ
Adding RStudio Workbench Users
RStudio Workbench uses Linux’s PAM authentication so any users accounts configured for Linux will be able to access RStudio Workbench. Users can be added for RStudio by the adduser Linux command.The following commands add a user “rstudio1” to Linux (and hence RStudio Workbench).
IMPORTANT: Each user must have a home directory associated with them otherwise they won’t be able to access RStudio since RStudio works on this home directory.
Verify Access
Verify access to RStudio Workbench for the newly added user:Go to ip:8787, enter user and pw for the newly created user, verify the user can be logged in to RStudio successfully.Troubleshoot for permission denied error on login:
Try adding the following line to /etc/sssd/sssd.conf (you may need to install sssd):
ad_gpo_map_service = +rstudio
Did you create the user with a specified home directory?
Make sure you didn’t configure the RStudio Workbench server to limit user access by user group (the default is to allow all; check /etc/rstudio/rserver.conf), or if so make sure your user is added to an allowed group.
Troubleshooting
After making any change to the RStudio Workbench config, always run the following for the change to take effect:
Premium Feature — This feature supports RStudio, which is part of the Professional and Enterprise Editions of LabKey Server. Learn more or contact LabKey.
After LabKey is configured to support either RStudio or RStudio Workbench it can be enabled in any folder where users want to be able to edit R Reports using RStudio.
Note: To edit R reports in RStudio, you must have Rlabkey v2.2.2 (or higher).
Within LabKey, open the R Report Builder. There are several paths:
Edit an existing R report (shown below).
Create a new report from a grid by selecting (Charts/Reports) > Create R Report.
From the Data Views web part, select Add Report > R Report.
Open the R Report Builder.
Click the Source tab and notice at the bottom there is an Edit in RStudio button.
Note: do not include slash characters (/) in your R report name.
When you click Edit in RStudio, you will see a popup giving the information that your report is now open for editing in RStudio. Click Go To RStudio to see it.
This may be in a hidden browser window or tab. This popup will remain in the current browser until you complete work in RStudio and click Edit in LabKey to return to the LabKey report editor.
Edit in RStudio
When you open a report to edit in RStudio, the editor interface will open. After making your changes, click the Save button (circled). Changes will be saved to LabKey.
Edit in RStudio Workbench
When you are using RStudio Workbench, if you have multiple sessions running, you will land on the home page. Select the session of interest and edit your report there.Remember to save your report in RStudio Workbench before returning to the LabKey Server browser window and clicking Edit in LabKey.
Premium Feature — This feature supports RStudio, which is part of the Professional and Enterprise Editions of LabKey Server. Learn more or contact LabKey.
This topic describes how users can export data from grids for use in their RStudio or RStudio Workbench environment. For guidance in configuring your server, first complete one of the following:
Select Export > RStudio. If you don't see this option, your server is not set up to use either RStudio or RStudio Workbench.
Enter a Variable Name and click Export to RStudio.
This will launch RStudio with your data loaded into the named variable.
Exporting to RStudio Workbench
When you are using RStudio Workbench and have multiple sessions running, when you click Export to RStudio you will first see your dashboard of open sessions. Select the session to which to export the data.
Premium Feature — This feature supports RStudio, which is part of the Professional and Enterprise Editions of LabKey Server. Learn more or contact LabKey.
This topic describes the process for initializing RStudio to enable the viewing of the LabKey schema within RStudio or RStudio Workbench. By default this option is enabled.
Note: To use this feature, you must have the latest version of Rlabkey.
On the Configure RStudio page for either type of integration, scroll down and confirm that the box for Initialize Schema Viewer is checked.
Use the LabKey Schema within RStudio
The RStudio viewer will now allow you to browse the LabKey schema. Select a project from the menu circled in this screenshot, then select a folder to see the schema it contains, and a schema to see the queries it contains.
Use LabKey Data with RStudio Shiny Apps
Once you have integrated either RStudio or RStudio Workbench with LabKey Server, you can create Shiny reports in RStudio that present LabKey data. This example shows an RStudio Shiny app giving user control of a live visualization of viral load over time.
Extend Initialization
An RStudio system administrator can add or overwrite variables and settings during initialization by including an R function in the Rprofile.site file which calls the hidden/named function "labkey.rstudio.extend". When performing this initialization, a one time "requestId" is used, rather than a more persistent api key.As an example, the following code introduces a simple call of this function which will print a 'Welcome' message (ignoring the self-signed certificate):
## define labkey.rstudio.extend function here, which will be called on RStudio R session initialization # ## Example: # labkey.rstudio.extend <- function() cat("nWelcome, LabKey user!nn") library("Rlabkey") labkey.acceptSelfSignedCerts()
RStudio executes the hidden function upon initialization:
LabKey Server provides rich tools for working with databases and SQL queries. By developing SQL queries you can:
Create filtered grid views of data.
Join data from different tables.
Group data and compute aggregates for each group.
Add a calculated column to a query.
Format how data is displayed using query metadata.
Create staging tables for building reports.
Special features include:
An intuitive table-joining syntax called lookups. Lookups use a convenient syntax of the form "Table.ForeignKey.FieldFromForeignTable" to achieve what would normally require a JOIN in SQL. For details see LabKey SQL Reference.
Parameterized SQL statements. Pass parameters to a SQL query via a JavaScript API.
How are SQL Queries processed?: LabKey Server parses the query, does security checks, and basically "cross-compiles" the query into the native dialect before passing it on to the underlying database.How can I tell what table a column came from?: If you want users to be able to tell the original source table for a column, use a label or description text for the original table field so that users can see that information when they hover over the grid header.
SQL queries are a powerful way to customize how you view and use data. In LabKey Server, queries are the main way to surface data from the database: using a query, you pick the columns you want to retrieve, and optionally apply any filters and sorts. SQL Queries are added to the database schema alongside the original, core tables. You can query one table at a time, or create a query that combines data from multiple tables. This topic describes how to create and modify queries using the user interface:
Queries also provide staging for reports: start with a base query and build a report on top of that query. Queries can be created through the graphical user interface (as shown in this topic) or through a file-based module.
SQL Resources
LabKey Server provides a number of mechanisms to simplify SQL syntax, covered in the topics linked here:
LabKey SQL: LabKey SQL is a SQL dialect that translates your queries into the native syntax of the SQL database underlying your server.
Lookups: Lookups join tables together using an intuitive syntax.
Query Metadata: Add additional properties to a query using metadata xml files, such as: column captions, relationships to other tables or queries, data formatting, and links.
The following step-by-step tutorial shows you how to create a SQL query and begin working with it.
In this example, we will create a query based on the Users table in the core schema.
Select > Go To Module > Query.
Learn more about using the query schema browser here: SQL Query Browser.
Open the core schema in the left hand pane and select the Users table.
Click the button Create New Query (in the top menu bar). This tells LabKey Server to create a query, and when you have selected a table first, by default it will be based on that table.
On the New Query page:
Provide a name for the query (in the field "What do you want to call the new query?"). Here we use "UsersQuery" though if that name is taken, you can use something else.
Confirm that the Users table is selected (in the field "Which query/table do you want this new query to be based on?")
Click Create and Edit Source.
LabKey Server will provide a "starter query" of the Users table: a basic SELECT statement for all of the fields in the table; essentially a duplicate of the original table, but giving you more editability. Typically, you would modify this "starter query" to fit your needs, changing the columns selected, adding WHERE clauses, JOINs to other tables, or substituting an entirely new SQL statement. For now, save the "starter query" unchanged.
Click Save & Finish.
The results of the query are displayed in a data grid, similar to the below, showing the users in your system.
Click core Schema to return to the query browser view of this schema.
Notice that your new query appears on the list alphabetically with an icon indicating it is a user-defined query.
Query Metadata
Like the original table, each query has accompanying XML that defines properties, or metadata, for the query. In this step we will add properties to the query by editing the accompanying XML. In particular we will:
Change the data type of the UserId column, making it a lookup into the Users table. By showing a clickable name instead of an integer value, we can make this column more human-readable and simultaneously connect it to more useful information.
Modify the way it is displayed in the grid. We will accomplish this by editing the XML directly.
Click Edit Metadata.
On the UserId row, click in the Data Type column where it shows the value Integer
From the dropdown, select User. This will create a lookup between your query and the Users table.
Click Apply.
Scroll down, click View Data, and click Yes, Save in the popup to confirm your changes.
Notice that the values in the User Id column are no longer integers, but text links with the user's display name shown instead of the previous integer value.
This reflects the fact that User Id is now a lookup into the Users table.
The display column for a lookup is (by default) the first text value in the target table. The actual integer value is unchanged. If you wish, you can expose the "User Id" column to check (use the grid customizer).
Click a value in the User Id column to see the corresponding record in the Users table.
This lookup is defined in the XML metadata document. Click back in your browser to return to the query, and let's see what the XML looks like.
Click core Schema to return to the Query Browser.
Click Edit Source and then select the XML Metadata tab.
The XML metadata will appear in a text editor. Notice the XML between the <fk>...</fk> tags. This tells LabKey Server to create a lookup (aka, a "foreign key") to the Users table in the core schema.
Next we will modify the XML directly to hide the "Display Name" column in our query. We don't need this column any longer because the User Id column already displays this information.
Add the following XML to the document, directly after the </column> tag (i.e directly before the ending </columns> tag). Each <column> tag in this section can customize a different column.
Click the Data tab to see the results without leaving the query editor (or saving the change you made).
Notice that the Display Name column is no longer shown.
Click the XML Metadata tab and now add the following XML immediately after the column section for hiding the display name. This section will display the Email column values in red.
Now that you have a SQL query, you can display it directly by using a query web part, or use it as the basis for a report, such as an R report or a visualization. It's important to note that nothing has changed about the underlying query (core.Users) on which your user-defined one (core.UsersQuery) is based. You could further define a new query based on the user-defined one as well. Each layer of query provides a new way to customize how your data is presented for different needs.
A schema is a named collection of tables and queries. The Query Browser, or Schema Browser, is the dashboard for browsing all the database data in a LabKey Server folder. It also provides access to key schema-related functionality. Using the schema browser, users with sufficient permissions can:
Browse the tables and queries
Add new SQL queries
Discover table/query relationships to help write new queries
Define external schemas to access new data
Generate scripts for bulk insertion of data into a new schema
Premium Feature AvailableWith Premium Editions of LabKey Server, administrators have the ability to trace query dependencies within a container or across folders. Learn more in this topic:
Authorized users can open the Query Schema Browser via one of these pathways:
> Go To Module > Query
> Developer Links > Schema Browser
The browser displays a list of the available schemas, represented with a folder icon, including external schemas and data sources you have added. Your permissions within the folder determine which tables and queries you can see here. Use the Show Hidden Schemas and Queries checkbox to see those that are hidden by default.Schemas live in a particular container (project or folder) on LabKey Server, but can be marked as inheritable, in which case they are accessible in child folders. Learn more about controlling schema heritability in child folders in this topic:Query Metadata
Built-In Tables vs. Module-Defined Queries vs. User-Defined Queries
Click the name of a schema to see the queries and tables it contains. User-defined, module-defined, and built-in queries are listed alphabetically, each category represented by a different icon. Checkboxes in the lower left can be used to control the set shown, making it easier to find a desired item on a long list.
Show Hidden Schemas and Queries
Show User-Defined Queries
Show Module-Defined Queries
Show Built-In Tables: Hard tables that are defined in the database, either by a user, module, or in code, but is not a query or in one of the above categories.
All Columns vs. Columns in the Default Grid View
You can browse columns by clicking on a particular table or query. The image below shows the column names of a user-defined query. In the Professional and Enterprise Editions of LabKey Server, you will also see a Dependency Report here.For a particular table or query, the browser shows two separate lists of columns:
All columns in this table: the top section shows all of the columns in the table/query.
Columns in your default view of this query: Your default grid view of the table/query may not contain all of the columns present in the table. It also may contain additional "columns" when there are lookups (or foreign keys) defined in the default view. See Step 1: Customize Your Grid View to learn about changing the contents of the "default" view.
Below these lists for Built-In Tables, the Indices of the table are listed.
Validate Queries
When you upgrade to a new version of LabKey, change hardware, or database software, you may want to validate your SQL queries. You can perform a validation check of your SQL queries by pressing the Validate Queries button, on the top row of buttons in the Query Schema Browser.Validation runs against all queries in the current folder and checks to see if the SQL queries parse and execute without errors.
To validate entire projects, initiate the scan from the project level and select Validate subfolders.
Check Include system queries for a much more comprehensive scan of internal system queries. Note that many warnings raised by this validation will be benign (such as queries against internal tables that are not exposed to end users).
Check Validate metadata and views box for a more thorough validation of both query metadata and saved views, including warnings of potentially invalid conditions, like autoincrement columns set userEditable=true, errors like invalid column names, or case errors which would cause problems for case-sensitive js. Again, many warnings raised in this scan will be benign.
Note: When running broad query validation and also using the experimental feature to Block Malicious Clients, you may trigger internal safeguards against "too many requests in a short period of time". If this occurs, the server may "lock you out" reporting a "Try again later" message in any browser. This block applies for an hour and can also be released if a site administrator turns off the experimental feature and/or clears the server cache.
Export Query Validation Results
Once query validation has been run, you can click Export to download the results, making it easier to parse and search.The exported Excel file lists the container, type (Error or Warning), Query, and Discrepancy for each item in the list.
View Raw Schema and Table Metadata
Links to View Raw Schema Metadata are provided for each schema, including linked and external schemas, giving you an easy way to confirm expected datasource, scope, and tables.Links to View Raw Table Metadata are provided for each built in table, giving details including indices, key details, and extended column properties.
Generate Schema Export / Migrate Data to Another Schema
If you wish to move data from one LabKey Server schema to another LabKey Server schema, you can do so by generating a migration script. The system will read the source schema and generate:
A set of tab-separated value (TSV) files, one for each table in the source. Each TSV file is packaged as a .tsv.gz file.
A script for importing these tables into a target schema. This script must be used as an update script included in a module.
Note that the script only copies data, it does not create the target schema itself. The target schema must already exist for the import script to work.To generate the TSV files and the associated script:
Go the Schema Browser: > Developer Links > Schema Browser.
Click Generate Schema Export.
Select the data source and schema name.
Enter a directory path where the script and TSV files will be written, for example: C:\temp\exports. This directory must already exist on your machine for the export to succeed.
Click Export.
The file artifacts will be written to the path you specified.
Field Descriptions
Field Name
Description
Source Data Source
The data source where the data to export resides.
Source Schema
The schema you want to export.
Target Schema
The schema where you want to import the data.
Path in Script
Optional. If you intend to place the import script and the data files in separate directories, specify a path so that the import script can find the data.
Output Directory
Directory on your local machine where the import script and the data files will be written. This directory must already exist on your machine, it will not be created for you.
The generated script consists of a series of bulkImport calls that open the .tsv.gz data files and insert them into the target schema, in this case the target schema is 'assaydata'.
Now you can re-import the data by adding the generated .sql script and .tsv.gz files to a module as a SQL upgrade script. For details on adding SQL scripts to modules, see Modules: SQL Scripts.
Creating a custom SQL query gives you the ability to flexibly present the data in a table in any way you wish using SQL features like calculated columns, aggregation, formatting, filtering, joins, and lookups.
The following steps guide you through creating a custom SQL query and view on a data table.
Create a Custom SQL Query
Select > Go To Module > Query.
From the schema list, select the schema that includes your data table of interest.
Optionally select the table on which you want to base the new query.
Click Create New Query.
In the field What do you want to call the new query?, enter a name for your new query.
Select the base query/table under Which query/table do you want this new query to be based on?.
Click Create and Edit Source.
LabKey Server will generate a default SQL query for the selected table.
Table and column names including spaces must be quoted. For readability you can specify a 'nickname' for the table ("PE" shown above) and use it in the query.
Click the Data tab to see the results (so far just the original table).
Return to the Source tab and click Save & Finish to save your query.
Refine the source of this query as desired in the SQL source editor.For example, you might calculate the average temperature per participant as shown here:
SELECT PE.ParticipantID, ROUND(AVG(PE.Temperature), 1) AS AverageTemp FROM "Physical Exam" PE GROUP BY PE.ParticipantID
Query Metadata: Edit query metadata on the XML Metadata tab.
LabKey SQL Reference
LabKey SQL
LabKey SQL is a SQL dialect that supports (1) most standard SQL functionality and (2)
provides extended functionality that is unique to LabKey, including:
Security. Before execution, all SQL queries are checked against
the user's security roles/permissions.
Lookup columns. Lookup columns use an intuitive syntax to access
data in other tables to achieve what would normally require a JOIN statement. For
example: "SomeTable.ForeignKey.FieldFromForeignTable" For details see Lookups. The special lookup column "Datasets" is injected into each study
dataset and provides a syntax shortcut when joining the current dataset to another
dataset that has compatible join keys. See example
usage.
Cross-folder querying. Queries can be scoped to folders broader than the current folder and can draw from tables in folders other than the current folder. See Cross-Folder Queries.
Parameterized SQL statements. The PARAMETERS
keyword lets you define parameters for a query. An associated API gives you
control over the parameterized query from JavaScript code. See Parameterized SQL Queries.
Pivot tables. The PIVOT...BY and PIVOT...IN expressions provide a syntax for creating
pivot tables. See Pivot Queries.
User-related functions. USERID() and ISMEMBEROF(groupid) let you
control query visibility based on the user's group membership.
Ontology-related functions.(Premium Feature) Access preferred terms and ontology concepts from SQL queries. See Ontology SQL.
Lineage-related functions.(Premium Feature) Access ancestors and descendants of samples and data class entities. See Lineage SQL Queries.
Annotations. Override some column metadata using SQL annotations. See Use SQL Annotations.
Aliases can be explicitly named using the AS keyword. Note that the AS
keyword is optional: the following select clauses both create an alias
called "Name":
SELECT LCASE(FirstName) AS Name
SELECT LCASE(FirstName) Name
Implicit aliases are automatically generated for expressions in the
SELECT list. In the query below, an output column named "Expression1"
is automatically created for the expression "LCASE(FirstName)":
SELECT LCASE(FirstName) FROM PEOPLE
ASCENDING, ASC
Return ORDER BY results in ascending value order. See the ORDER BY section for troubleshooting notes.
ORDER BY Weight ASC
CAST(AS)
CAST(R.d AS VARCHAR)
Defined valid datatype keywords which can be used as cast/convert
targets, and to what java.sql.Types name each keyword maps. Keywords are
case-insensitive.
BIGINT
BINARY
BIT
CHAR
DECIMAL
DATE
DOUBLE
FLOAT
GUID
INTEGER
LONGVARBINARY
LONGVARCHAR
NUMERIC
REAL
SMALLINT
TIME
TIMESTAMP
TINYINT
VARBINARY
VARCHAR
Examples:
CAST(TimeCreated AS DATE)
CAST(WEEK(i.date) as INTEGER) as WeekOfYear,
DESCENDING, DESC
Return ORDER BY results in descending value order. See the ORDER BY section for troubleshooting notes.
ORDER BY Weight DESC
DISTINCT
Return distinct, non duplicate, values.
SELECT DISTINCT Country
FROM Demographics
EXISTS
Returns a Boolean value based on a subquery. Returns TRUE if at least one
row is returned from the subquery.
The following example returns any plasma samples which have been assayed with a
score greater than 80%. Assume that ImmuneScores.Data.SpecimenId is a lookup
field (aka foreign key) to Plasma.Name.
SELECT Plasma.Name
FROM Plasma
WHERE EXISTS
(SELECT *
FROM assay.General.ImmuneScores.Data
WHERE SpecimenId = Plasma.Name
AND ScorePercent > .8)
FALSE
FROM
The FROM clause in LabKey SQL must contain at least one table. It can also
contain JOINs to other tables. Commas are supported in the FROM clause:
FROM TableA, TableB
WHERE TableA.x = TableB.x
Nested joins are supported in the FROM clause:
FROM TableA LEFT JOIN (TableB INNER JOIN TableC ON
...) ON...
To refer to tables in LabKey folders other than the current folder, see
Cross-Folder Queries.
GROUP BY
Used with aggregate functions to group the results. Defines the "for
each" or "per". The example below returns the number of records "for
each" participant:
SELECT ParticipantId, COUNT(Created) "Number of
Records"
FROM "Physical Exam"
GROUP BY ParticipantId
HAVING
Used with aggregate functions to limit the results. The following
example returns participants with 10 or more records in the Physical Exam
table:
SELECT ParticipantId, COUNT(Created) "Number of Records"
FROM "Physical Exam"
GROUP BY ParticipantId
HAVING COUNT(Created) > 10
HAVING can be used without a GROUP BY clause, in which case all selected rows
are treated as a single group for aggregation purposes.
JOIN,
RIGHT JOIN,
LEFT JOIN,
FULL JOIN,
CROSS JOIN
Example:
SELECT *
FROM "Physical Exam"
FULL JOIN "Lab Results"
ON "Physical Exam".ParticipantId = "Lab
Results".ParticipantId
LIMIT
Limits the number or records returned by the query. The following
example returns the 10 most recent records:
SELECT *
FROM "Physical Exam"
ORDER BY Created DESC LIMIT 10
NULLIF(A,B)
Returns NULL if A=B, otherwise returns A.
ORDER BY
One option for sorting query results. It may produce unexpected results when dataregions or views also have sorting applied, or when using an expression in the ORDER BY clause, including an expression like table.columnName. If you can instead use a sort on the custom view or via, the API, those methods are preferred (see Troubleshooting note below).
For best ORDER BY results, be sure to a) SELECT the columns on which you are sorting, b) sort on the SELECT column, not on an expression. To sort on an expression, include the expression in the SELECT (hidden if desired) and sort by the alias of the expression. For example:
SELECT A, B, A+B AS C @hidden ... ORDER BY C
...is preferable to:
SELECT A, B ... ORDER BY A+B
Use ORDER BY with LIMIT to improve performance:
SELECT ParticipantID,
Height_cm AS Height
FROM "Physical Exam"
ORDER BY Height DESC LIMIT 5
Troubleshooting: "Why is the ORDER BY clause not working as
expected?"
1. Check to ensure you are sorting by a SELECT column (preferred) or an alias of an expression. Syntax like including the table name (i.e. ...ORDER BY table.columnName ASC) is an expression and should be aliased in the SELECT statement instead (i.e. SELECT table.columnName AS C ... ORDER BY C
2. When authoring queries in LabKey SQL, the query is typically processed as
a subquery within a parent query. This parent query may apply it's own
sorting overriding the ORDER BY clause in the subquery. This parent "view layer" provides
default behavior like pagination, lookups, etc. but may also unexpectedly apply an additional sort.
Two recommended solutions for more predictable sorting:
(A) Define the sort in the parent query using the grid view customizer. This may involve adding a new named view of that query to use as your parent query.
(B) Use the "sort" property in the
selectRows API call.
PARAMETERS
Queries can declare parameters using the PARAMETERS keyword. Default values
data types are supported as shown below:
PARAMETERS (X INTEGER DEFAULT 37) SELECT * FROM "Physical Exam" WHERE Temp_C = X
Parameter names will override any unqualified table column with the same
name. Use a table qualification to disambiguate. In the example
below, R.X refers to the column while X refers to the parameter:
PARAMETERS(X INTEGER DEFAULT 5) SELECT * FROM Table R WHERE R.X = X
Supported data types for parameters are: BIGINT, BIT, CHAR, DECIMAL,
DOUBLE, FLOAT, INTEGER, LONGVARCHAR, NUMERIC, REAL, SMALLINT, TIMESTAMP,
TINYINT, VARCHAR
Parameter values can be passed via JavaScript API calls to the query.
For details see Parameterized SQL Queries.
PIVOT/PIVOT...BY/PIVOT...IN
Re-visualize a table by rotating or "pivoting" a portion of it, essentially
promoting cell data to column headers. See Pivot
Queries for details and examples.
SELECT
SELECT queries are the only type of query that can currently be written in
LabKey SQL. Sub-selects are allowed both as an expression, and in the
FROM clause.
Aliases are automatically generated for expressions after SELECT.
In the query below, an output column named "Expression1" is automatically
generated for the expression "LCASE(FirstName)":
SELECT LCASE(FirstName) FROM...
TRUE
UNION, UNION ALL
The UNION clause is the same as standard SQL. LabKey SQL supports
UNION in subqueries.
VALUES ... AS
A subset of VALUES syntax is supported. Generate a "constant table" by
providing a parenthesized list of expressions for each row in the table.
The lists must all have the same number of elements and corresponding
entries must have compatible data types. For example:
VALUES (1, 'one'), (2, 'two'), (3, 'three') AS t;
You must provide the alias for the result ("AS t" in the above), aliasing
column names is not supported. The column names will be 'column1',
'column2', etc.
WHERE
Filter the results for certain values. Example:
SELECT *
FROM "Physical Exam"
WHERE YEAR(Date) = 2010
WITH
Define a "common table expression" which functions like a subquery or
inline view table. Especially useful for recursive queries.
Usage Notes: If there are UNION clauses that do not reference the common
table expression (CTE) itself, the server interprets them as normal UNIONs.
The first subclause of a UNION may not reference the CTE. The CTE may only
be referenced once in a FROM clause or JOIN clauses within the UNION. There
may be multiple CTEs defined in the WITH. Each may reference the previous
CTEs in the WITH. No column specifications are allowed in the WITH (as some
SQL versions allow).
Exception Behavior: Testing indicates that PostgreSQL does not provide
an exception to LabKey Server for a non-ending, recursive CTE
query. This can cause the LabKey Server to wait indefinitely for the query
to complete.
A non-recursive example:
WITH AllDemo AS
(
SELECT *
FROM "/Studies/Study A/".study.Demographics
UNION
SELECT *
FROM "/Studies/Study B/".study.Demographics
)
SELECT ParticipantId from AllDemo
A recursive example: In a table that holds parent/child information, this query returns all of the children and grandchildren (recursively down the generations), for a given "Source" parent.
PARAMETERS
(
Source VARCHAR DEFAULT NULL
)
WITH Derivations AS
(
-- Anchor Query. User enters a 'Source' parent
SELECT Item, Parent
FROM Items
WHERE Parent = Source
UNION ALL
-- Recursive Query. Get the children, grandchildren, ... of the source parent
SELECT i.Item, i.Parent
FROM Items i INNER JOIN Derivations p
ON i.Parent = p.Item
)
SELECT * FROM Derivations
Constants
The following constant values can be used in LabKey SQL queries.
Constant
Description
CAST('Infinity' AS DOUBLE)
Represents positive infinity.
CAST('-Infinity' AS DOUBLE)
Represents negative infinity.
CAST('NaN' AS DOUBLE)
Represents "Not a number".
TRUE
Boolean value.
FALSE
Boolean value.
Operators
Operator
Description
String Operators
Note that strings are delimited with single quotes. Double quotes are used for column and table names containing spaces.
||
String concatenation. For example:
SELECT ParticipantId,
City || ', ' || State AS CityOfOrigin
FROM Demographics
If any argument is null, the || operator will return a null string. To handle this, use COALESCE with an empty string as it's second argument, so that the other || arguments will be returned:
City || ', ' || COALESCE (State, '')
LIKE
Pattern matching. The entire string must match the given pattern. Ex: LIKE 'W%'.
NOT LIKE
Negative pattern matching. Will return values that do not match a given pattern. Ex: NOT LIKE 'W%'
Arithmetic Operators
+
Add
-
Subtract
*
Multiply
/
Divide
Comparison operators
=
Equals
!=
Does not equal
<>
Does not equal
>
Is greater than
<
Is less than
>=
Is greater than or equal to
<=
Is less than or equal to
IS NULL
Is NULL
IS NOT NULL
Is NOT NULL
BETWEEN
Between two values. Values can be numbers, strings or dates.
IN
Example: WHERE City IN ('Seattle', 'Portland')
NOT IN
Example: WHERE City NOT IN ('Seattle', 'Portland')
Bitwise Operators
&
Bitwise AND
|
Bitwise OR
^
Bitwise exclusive OR
Logical Operators
AND
Logical AND
OR
Logical OR
NOT
Example: WHERE NOT Country='USA'
Operator Order of Precedence
Order of Precedence
Operators
1
- (unary) , + (unary), CASE
2
*, / (multiplication, division)
3
+, -, & (binary plus, binary
minus)
4
& (bitwise and)
5
^ (bitwise xor)
6
| (bitwise or)
7
|| (concatenation)
8
<, >, <=, >=, IN, NOT IN, BETWEEN, NOT
BETWEEN, LIKE, NOT LIKE
9
=, IS, IS NOT, <>, !=
10
NOT
11
AND
12
OR
Aggregate Functions - General
Function
Description
COUNT
The special syntax COUNT(*) is supported as of LabKey v9.2.
MIN
Minimum
MAX
Maximum
AVG
Average
SUM
Sum
GROUP_CONCAT
An aggregate function, much like MAX, MIN, AVG, COUNT, etc. It can be used
wherever the standard aggregate functions can be used, and is subject to the
same grouping rules. It will return a string value which is comma-separated list
of all of the values for that grouping. A custom separator, instead of the
default comma, can be specified. Learn more here.
The example below specifies a semi-colon as the separator:
SELECT Participant, GROUP_CONCAT(DISTINCT Category,
';') AS CATEGORIES FROM SomeSchema.SomeTable
To use a line-break as the separator, use the following:
SELECT Participant, GROUP_CONCAT(DISTINCT Category,
chr(10)) AS CATEGORIES FROM SomeSchema.SomeTable
stddev(expression)
Standard deviation
stddev_pop(expression)
Population standard deviation of the input values.
variance(expression)
Historical alias for var_samp.
var_pop(expression)
Population variance of the input values (square of the population standard
deviation).
median(expression)
The 50th percentile of the values submitted.
Aggregate Functions - PostgreSQL Only
Function
Description
bool_and(expression)
Aggregates boolean values. Returns true if all values are true and false if
any are false.
bool_or(expression)
Aggregates boolean values. Returns true if any values are true and false if
all are false.
bit_and(expression)
Returns the bitwise AND of all non-null input values, or null if none.
bit_or(expression)
Returns the bitwise OR of all non-null input values, or null if none.
every(expression)
Equivalent to bool_and(). Returns true if all values are true and false if
any are false.
corr(Y,X)
Correlation coefficient.
covar_pop(Y,X)
Population covariance.
covar_samp(Y,X)
Sample covariance.
regr_avgx(Y,X)
Average of the independent variable: (SUM(X)/N).
regr_avgy(Y,X)
Average of the dependent variable: (SUM(Y)/N).
regr_count(Y,X)
Number of non-null input rows.
regr_intercept(Y,X)
Y-intercept of the least-squares-fit linear equation determined by the
(X,Y) pairs.
regr_r2(Y,X)
Square of the correlation coefficient.
regr_slope(Y,X)
Slope of the least-squares-fit linear equation determined by the (X,Y)
pairs.
regr_sxx(Y,X)
Sum of squares of the independent variable.
regr_sxy(Y,X)
Sum of products of independent times dependent variable.
regr_syy(Y,X)
Sum of squares of the dependent variable.
stddev_samp(expression)
Sample standard deviation of the input values.
var_samp(expression)
Sample variance of the input values (square of the sample standard
deviation).
SQL Functions - General
Many of these functions are similar to standard SQL functions -- see the JBDC escape syntax
documentation for additional information.
Function
Description
abs(value)
Returns the absolute value.
acos(value)
Returns the arc cosine.
age(date1, date2)
Supplies the difference in age between the two dates, calculated in
years.
age(date1, date2, interval)
The interval indicates the unit of age measurement, either SQL_TSI_MONTH
or SQL_TSI_YEAR.
age_in_months(date1, date2)
Behavior is undefined if date2 is before date1.
age_in_years(date1, date2)
Behavior is undefined if date2 is before date1.
asin(value)
Returns the arc sine.
atan(value)
Returns the arc tangent.
atan2(value1, value2)
Returns the arctangent of the quotient of two values.
case
CASE can be used to test various conditions and return various results based on the test. You can use either simple CASE or searched CASE syntax. In the following examples "value#" indicates a value to match against, where "test#" indicates a boolean expression to evaluate. In the "searched" syntax, the first test expression that evaluates to true will determine which result is returned. Note that the LabKey SQL parser sometimes requires the use of additional parentheses within the statement.
CASE (value) WHEN (value1) THEN (result1) ELSE
(result2) END
CASE (value) WHEN (value1) THEN (result1) WHEN (value2) THEN (result2) ELSE
(resultDefault) END
CASE WHEN (test1) THEN (result1) ELSE (result2)
END
CASE WHEN (test1) THEN (result1) WHEN (test2) THEN (result2) WHEN (test3) THEN (result3) ELSE (resultDefault)
END
Example:
SELECT "StudentName",
School,
CASE WHEN (Division = 'Grades 3-5') THEN (Scores.Score*1.13)
ELSE Score END AS AdjustedScore,
Division
FROM Scores
ceiling(value)
Rounds the value up.
coalesce(value1,...,valueN)
Returns the first non-null value in the argument list. Use to set default
values for display.
concat(value1,value2)
Concatenates two values.
contextPath()
Returns the context path starting with “/” (e.g.
“/labkey”). Returns the empty string if there is no current
context path. (Returns VARCHAR.)
cos(radians)
Returns the cosine.
cot(radians)
Returns the cotangent.
curdate()
Returns the current date.
curtime()
Returns the current time
dayofmonth(date)
Returns the day of the month (1-31) for a given date.
dayofweek(date)
Returns the day of the week (1-7) for a given date. (Sun=1 and Sat=7)
dayofyear(date)
Returns the day of the year (1-365) for a given date.
degrees(radians)
Returns degrees based on the given radians.
exp(n)
Returns Euler's number e raised to the nth power.
e = 2.71828183
floor(value)
Rounds down to the nearest integer.
folderName()
LabKey SQL extension function. Returns the name
of the current folder, without beginning or trailing "/". (Returns
VARCHAR.)
folderPath()
LabKey SQL
extension function. Returns the current folder path (starts with
“/”, but does not end with “/”). The root returns
“/”. (Returns VARCHAR.)
greatest(a, b, c, ...)
Returns the greatest value from the list expressions provided. Any number
of expressions may be used. The expressions must have the same data type,
which will also be the type of the result. The LEAST() function is similar,
but returns the smallest value from the list of expressions. GREATEST() and
LEAST() are not implemented for SAS databases.
When NULL values appear in the list of expressions, different database
implementations as follows:
- PostgreSQL & MS SQL Server ignore NULL values in the arguments, only
returning NULL if all arguments are NULL.
- Oracle and MySql return NULL if any one of the arguments are
NULL. Best practice: wrap any potentially nullable arguments in coalesce()
or ifnull() and determine at the time of usage if NULL should be treated as
high or low.
Example:
SELECT greatest(score_1, score_2, score_3) As HIGH_SCORE
FROM MyAssay
hour(time)
Returns the hour for a given date/time.
ifdefined(column_name)
IFDEFINED(NAME) allows queries to reference columns that may not be
present on a table. Without using IFDEFINED(), LabKey will raise a SQL parse
error if the column cannot be resolved. Using IFDEFINED(), a column that cannot
be resolved is treated as a NULL value. The IFDEFINED() syntax is useful for
writing queries over PIVOT queries or assay tables where columns may be added
or removed by an administrator.
ifnull(testValue, defaultValue)
If testValue is null, returns the defaultValue. Example:
IFNULL(Units,0)
isequal
LabKey SQL extension
function. ISEQUAL(a,b) is equivalent to (a=b OR (a
IS NULL AND b IS NULL))
ismemberof(groupid)
LabKey SQL extension function. Returns true if the current
user is a member of the specified group.
javaConstant(fieldName)
LabKey SQL extension function. Provides access to public
static final variable values. For details see LabKey SQL Utility Functions.
lcase(string)
Convert all characters of a string to lower case.
least(a, b, c, ...)
Returns the smallest value from the list expressions provided. For more
details, see greatest() above.
left(string, integer)
Returns the left side of the string, to the given number of characters.
Example: SELECT LEFT('STRINGVALUE',3) returns 'STR'
Returns the location of the first occurrence of substring within
string. startIndex provides a starting position to begin the
search.
log(n)
Returns the natural logarithm of n.
log10(n)
Base base 10 logarithm on n.
lower(string)
Convert all characters of a string to lower case.
ltrim(string)
Trims white space characters from the left side of the string. For example:
LTRIM(' Trim String')
minute(time)
Returns the minute value for the given time.
mod(dividend, divider)
Returns the remainder of the division of dividend by divider.
moduleProperty(module name, property name)
LabKey
SQL extension function. Returns a module property, based on the
module and property names. For details see LabKey
SQL Utility Functions.
month(date)
Returns the month value (1-12) of the given date.
monthname(date)
Return the month name of the given date.
now()
Returns the system date and time.
overlaps
LabKey SQL extension
function. Supported only when PostgreSQL is installed as the
primary database.
SELECT OVERLAPS (START1, END1, START2, END2) AS COLUMN1 FROM
MYTABLE
The LabKey SQL syntax above is translated into the following PostgreSQL
syntax:
SELECT (START1, END1) OVERLAPS (START2, END2) AS COLUMN1 FROM
MYTABLE
pi()
Returns the value of π.
power(base, exponent)
Returns the base raised to the power of
the exponent. For example, power(10,2) returns 100.
quarter(date)
Returns the yearly quarter for the given date where the 1st quarter = Jan
1-Mar 31, 2nd quarter = Apr 1-Jun 30, 3rd quarter = Jul 1-Sep 30, 4th
quarter = Oct 1-Dec 31
radians(degrees)
Returns the radians for the given degrees.
rand(), rand(seed)
Returns a random number between 0 and 1.
repeat(string, count)
Returns the string repeated the given number of times. SELECT
REPEAT('Hello',2) returns 'HelloHello'.
round(value, precision)
Rounds the value to the specified number of decimal places.
ROUND(43.3432,2) returns 43.34
rtrim(string)
Trims white space characters from the right side of the string. For
example: RTRIM('Trim String ')
second(time)
Returns the second value for the given time.
sign(value)
Returns the sign, positive or negative, for the given value.
sin(value)
Returns the sine for the given value.
startswith(string, prefix)
Tests to see if the string starts with the specified
prefix. For example, STARTSWITH('12345','2') returns FALSE.
sqrt(value)
Returns the square root of the value.
substring(string, start, length)
Returns a portion of the string as specified by the start
location (1-based) and length (number of characters). For example,
substring('SomeString', 1,2) returns the string 'So'.
tan(value)
Returns the tangent of the value.
timestampadd(interval, number_to_add, timestamp)
Adds an interval to the given timestamp value. The
interval value must be surrounded by quotes. Possible values for
interval:
As a workaround, use the 'age' functions defined above.
truncate(numeric value, precision)
Truncates the numeric value to the precision
specified. This is an arithmetic truncation, not a string truncation.
TRUNCATE(123.4567,1) returns 123.4
TRUNCATE(123.4567,2) returns 123.45
TRUNCATE(123.4567,-1) returns 120.0
May require an explict CAST into NUMERIC, as LabKey SQL does not check data types for function arguments.
SELECT
PhysicalExam.Temperature,
TRUNCATE(CAST(Temperature AS NUMERIC),1) as truncTemperature
FROM PhysicalExam
ucase(string), upper(string)
Converts all characters to upper case.
userid()
LabKey SQL extension function. Returns the userid, an
integer, of the logged in user.
username()
LabKey SQL extension function. Returns the current user
display name. VARCHAR
version()
LabKey SQL extension function. Returns the current schema
version of the core module as a numeric with four decimal places. For example: 20.0070
week(date)
Returns the week value (1-52) of the given date.
year(date)
Return the year of the given date. Assuming the system date is March
4 2023, then YEAR(NOW()) return 2023.
SQL Functions - PostgreSQL Specific
LabKey SQL supports the following PostgreSQL functions.
See the PostgreSQL
docs for usage details.
PostgreSQL Function
Docs
ascii(value)
Returns the ASCII code
of the first character of value.
btrim(value,
trimchars)
Removes characters in trimchars from the start and end of
string. trimchars defaults to white space.
BTRIM(' trim ') returns TRIM
BTRIM('abbatrimabtrimabba', 'ab') returns trimabtrim
character_length(value), char_length(value)
Returns the number of characters in value.
chr(integer_code)
Returns the character with the given integer_code.
CHR(70) returns F
concat_ws(sep text, val1 "any" [, val2 "any" [,...]]) -> text
Concatenates all but the first argument, with separators. The first argument is used as the separator string, and should not be NULL. Other NULL arguments are ignored. See the PostgreSQL docs.
LabKey SQL supports the following PostgreSQL JSON and JSONB operators and functions. Note that LabKey SQL does not natively understand arrays and some other features, but it may still be possible to use the functions that expect them.
See the PostgreSQL docs for usage details.
PostgreSQL Operators and Functions
Docs
->, ->>, #>, #>>, @>, <@, ?, ?|, ?&, ||, -, #-
LabKey SQL supports these operators via a pass-through function, json_op. The function's first argument is the operator's first operand. The first second is the operator, passed as a string constant. The function's third argument is the second operand. For example, this Postgres SQL expression:
a_jsonb_column --> 2
can be represented in LabKey SQL as:
json_op(a_jsonb_column, '-->', 2)
parse_json, parse_jsonb
Casts a text value to a parsed JSON or JSONB data type. For example,
'{"a":1, "b":null}'::jsonb
or
CAST('{"a":1, "b":null}' AS JSONB)
can be represented in LabKey SQL as:
parse_jsonb('{"a":1, "b":null}')
to_json, to_jsonb
Converts a value to the JSON or JSONB data type. Will treat a text value as a single JSON string value
array_to_json
Converts an array value to the JSON data type.
row_to_json
Converts a scalar (simple value) row to JSON. Note that LabKey SQL does not support the version of this function that will convert an entire table to JSON. Consider using "to_jsonb()" instead.
json_build_array, jsonb_build_array
Build a JSON array from the arguments
json_build_object, jsonb_build_object
Build a JSON object from the arguments
json_object, jsonb_object
Build a JSON object from a text array
json_array_length, jsonb_array_length
Return the length of the outermost JSON array
json_each, jsonb_each
Expand the outermost JSON object into key/value pairs. Note that LabKey SQL does not support the table version of this function. Usage as a scalar function like this is supported:
SELECT json_each('{"a":"foo", "b":"bar"}') AS Value
json_each_text, jsonb_each_text
Expand the outermost JSON object into key/value pairs into text. Note that LabKey SQL does not support the table version of this function. Usage as a scalar function (similar to json_each) is supported.
json_extract_path, jsonb_extract_path
Return the JSON value referenced by the path
json_extract_path_text, jsonb_extract_path_text
Return the JSON value referenced by the path as text
Insert a value within a JSON object at a given path
jsonb_pretty
Format a JSON object as indented text
jsonb_set
Set the value within a JSON object for a given path. Strict, i.e. returns NULL on NULL input.
jsonb_set_lax
Set the value within a JSON object for a given path. Not strict; expects third argument to specify how to treat NULL input (one of 'raise_exception', 'use_json_null', 'delete_key', or 'return_target').
jsonb_path_exists, jsonb_path_exists_tz
Checks whether the JSON path returns any item for the specified JSON value. The "_tz" variant is timezone aware.
jsonb_path_match, jsonb_path_match_tz
Returns the result of a JSON path predicate check for the specified JSON value. The "_tz" variant is timezone aware.
jsonb_path_query, jsonb_path_query_tz
Returns all JSON items returned by the JSON path for the specified JSON value. The "_tz" variant is timezone aware.
jsonb_path_query_array, jsonb_path_query_array_tz
Returns as an array, all JSON items returned by the JSON path for the specified JSON value. The "_tz" variant is timezone aware.
jsonb_path_query_first, jsonb_path_query_first_tz
Returns the first JSON item returned by the JSON path for the specified JSON value. The "_tz" variant is timezone aware.
SQL Functions - MS SQL Server Specific
LabKey SQL supports the following SQL Server functions. Note that this functionality is only available to existing Premium Edition subscribers already using Microsoft SQL Server.
See the SQL Server docs for usage details.
MS SQL Server Function
Description
ascii(value)
Returns the ASCII code of the first character of value.
char(int), chr(int)
Returns an character for the specified ascii code int.
Returns the position of expressionToFind in expressionToSearch,
starting the search at position index.
concat_ws(sep text, val1 "any" [, val2 "any" [,...]]) -> text
Concatenates all but the first argument, with separators. The first argument is used as the separator string, and should not be NULL. Other NULL arguments are ignored.
concat_ws(',', 'abcde', 2, NULL, 22) → abcde,2,22
difference(string,string)
Returns the difference between the soundex values of two expressions as an
integer.
See the MS SQL docs.
isnumeric(expression)
Determines whether an expression is a valid numeric type. See the MS SQL
docs.
len(string)
Returns the number of characters in string. Trailing white space
is excluded.
patindex(pattern,string)
Returns the position of the first occurrence of pattern in
string. See the MS SQL docs.
Inserts replaceWith into string. Deletes the
specified length of characters in string at the
start position and then inserts replaceWith. See the MS
SQL docs.
General Syntax
Syntax Item
Description
Case Sensitivity
Schema names, table names, column names, SQL keywords, function names are
case-insensitive in LabKey SQL.
Comments
Comments that use the standard SQL syntax can be included in queries. '--'
starts a line comment. Also, '/* */' can surround a comment block:
-- line comment 1
-- line comment 2
/* block comment 1
block comment 2 */
SELECT ...
Identifiers
Identifiers in LabKey SQL may be quoted using double quotes. (Double quotes
within an identifier are escaped with a second double quote.)
SELECT "Physical Exam".*
...
Lookups
Lookups columns reference data in other tables. In SQL terms, they
are foreign key columns. See Lookups for details on creating lookup columns. Lookups
use a convenient syntax of the form
"Table.ForeignKey.FieldFromForeignTable" to achieve what would normally
require a JOIN in SQL. Example:
Issues.AssignedTo.DisplayName
String Literals
String literals are quoted with single quotes ('). Within a single quoted
string, a single quote is escaped with another single quote.
SELECT *
FROM TableName WHERE FieldName =
'Jim''s Item'
Date/Time Literals
Date and Timestamp (Date&Time) literals can be specified using the
JDBC escape syntax
Lookups are an intuitive table linking syntax provided to simplify data integration and SQL queries. They represent foreign key relationships between tables, and once established, can be used to "expose" columns from the "target" of the lookup in the source table or query.All lookups are secure: before execution, all references in a query are checked against the user's security role/permissions, including lookup target tables.
This topic describes using lookups from SQL. Learn about defining lookups via the user interface in this topic: Lookup Columns.
Lookup SQL Syntax
Lookups have the general form:
Table.ForeignKey.FieldFromForeignTable
Examples
A lookup effectively creates a join between the two tables. You can follow this join to show columns in the looked-up table. For instance if "tableA" has a "Name", plus a field "mylookup" that is a lookup to "tableB" and "tableB" has a column called "something", you can use this syntax:
SELECT tableA.Name, tableA.mylookup.something, FROM tableA
You can also use a shorthand syntax, omitting the "tableA" part of the lookup reference. The "." in something like "mylookup.name" does a left outer join for you, rather than requiring the full "tableA.mylookup.something". This can be useful in (for example) using foreign key relationships in where clauses such as the following:
SELECT * FROM tableA WHERE mylookup.something = 'A01'
Datasets Lookup
The following query uses the "Datasets" special column that is present in every study dataset table to lookup values in the Demographics table (another dataset in the same study), joining them to the PhysicalExam table.
SELECT PhysicalExam.ParticipantId, PhysicalExam.date, PhysicalExam.Height_cm, Datasets.Demographics.Gender AS GenderLookup, FROM PhysicalExam
It replaces the following JOIN statement.
SELECT PhysicalExam.ParticipantId, PhysicalExam.date, PhysicalExam.Height_cm, Demographics.Gender AS GenderLookup FROM PhysicalExam INNER JOIN Demographics ON PhysicalExam.ParticipantId = Demographics.ParticipantId
Lookup From a List
The following expressions show the Demographics table looking up values in the Languages table.
SELECT Demographics.ParticipantId, Demographics.StartDate, Demographics.Language.LanguageName, Demographics.Language.TranslatorName, Demographics.Language.TranslatorPhone FROM Demographics
It replaces the following JOIN statement.
SELECT Demographics.ParticipantId, Demographics.StartDate, Languages.LanguageName, Languages.TranslatorName, Languages.TranslatorPhone FROM Demographics LEFT OUTER JOIN lists.Languages ON Demographics.Language = Languages.LanguageId;
Lookup User Last Name
The following lookup expression shows the Issues table looking up data in the Users table, retrieving the Last Name.
Issues.UserID.LastName
Discover Lookup Column Names
To discover lookup relationships between tables:
Go to > Developer Links > Schema Browser.
Select a schema and table of interest.
Browse lookup fields by clicking the icon next to a column name which has a lookup table listed.
In the image below, the column study.Demographics.Language looks up the lists.Languages table joining on the column LanguageId.
Available columns in the Languages table are listed (in the red box). To reference these columns in a SQL query, use the lookup syntax: Demographics.Language."col_in_lookup_table", i.e. Demographics.Language.TranslatorName, Demographics.Language.TranslatorPhone, etc.
Note that the values are shown using the slash-delimited syntax, which is used in the selectRows API. API documentation is available here:
Before lookup columns can be used, they need to be added to the definition of a dataset/list/query. The process of setting up lookup relationships in the field editor is described here: Lookup Columns.
Creating a Lookup to a Non-Primary-Key Field
When you select a schema, all tables with a primary key of the type matching the field you are defining are listed by default. If you want to create a lookup to a column that is not the primary key, you must first be certain that it is unique (i.e. could be a key), then take an additional step to mark the desired lookup field as a key.You also use the steps below to expose a custom SQL query as the target of a lookup.
First create a simple query to wrap your list:
SELECT * FROM list.MyList
You could also add additional filtering, joins, or group by clauses as needed.
Next, annotate the query with XML metadata to mark the desired target column as "isKeyField". On the XML Metadata tab of the query editor, enter:
LabKey SQL provides the function extensions covered in this topic to help Java module developers and others writing custom LabKey queries access various properties.
Returns a module property, based on the module and property names. Arguments are strings, so use single quotes not double.moduleProperty values can be used in place of a string literal in a query, or to supply a container path for Queries Across Folders.Examples
moduleProperty('EHR','EHRStudyContainer')
You can use the virtual "Site" schema to specify a full container path, such as '/home/someSubfolder' or '/Shared':
SELECT * FROM Site.{substitutePath moduleProperty('EHR','EHRStudyContainer')}.study.myQuery
The virtual "Project." schema similarly specifies the name of the current project.
Provides access to public static final variable values. The argument value should be a string.Fields must be either be on classes in the java.lang package, or tagged with the org.labkey.api.query.Queryable annotation to indicate they allow access through this mechanism. All values will be returned as if they are of type VARCHAR, and will be coerced as needed. Other fields types are not supported.Examples
Tables and queries can have associated XML files that carry additional metadata information about the columns in the query. Example uses of query metadata include:
Column titles and data display formatting
Add URLs to data in cells or lookups to other tables
Add custom buttons that navigate to other pages or call JavaScript methods, or disable the standard buttons
Color coding for values that fall within a numeric range
Create an alias field, a duplicate of an existing field to which you can apply independent metadata properties
Topics
You can edit or add to this metadata using a variety of options:
Another way to customize query metadata is using an XML file in a module. Learn more in this topic: Modules: Query Metadata
Edit Metadata Using the User Interface
The metadata editor offers a subset of the features available in the field editor and works in the same way offering fields with properties, configuration options, and advanced settings.
Open the schema browser via > Go To Module > Query.
Select an individual schema and query/table in the browser and click Edit Metadata.
Note that there are some settings that cannot be edited via metadata override and are thus inactive in the field editor.
You also cannot delete fields here, so will not see the icon.
Expand a field with .
To change a column's displayed title, edit the Label property.
In the image above, the displayed text for the column has been changed to include special characters that could cause issues if used in the field name.
Other metadata elements and attributes are listed in the tableInfo.xsd schema available in the XML Schema Reference.
Note that it is only possible to add/alter references to metadata entities that already exist in your query. For example, you can edit the "columnTitle" (aka the "Label" in the query designer) because this merely changes the string that provides the display name of the field. However, you cannot edit the "columnName" because this entity is the reference to a column in your query. Changing "columnName" breaks that reference.
For additional information about query metadata and the order of operations, see Modules: Query Metadata.
XML Metadata Filters
The filtering expressions available for use in XML metadata are listed below.
XML operator
Description
eq
Equals
dateeq
Equals for date fields
dateneq
Does not equal, for date fields
datelt
Less than, for date fields
datelte
Less than or equal to, for date fields
dategt
Greater than, for date fields
dategte
Greater than or equal to, for date fields
neq
Does not equal
neqornull
Does not equal or is null
isblank
Is blank
isnonblank
Is not blank
gt
Greater than
lt
Less than
gte
Greater than or equal to
lte
Less than or equal to
between
Between two provided values
notbetween
Not between two provided values
contains
Contains a provided substring
doesnotcontain
Does not contain a provided substring
doesnotstartwith
Does not start with a provided substring
startswith
Starts with a provided substring
in
Equals one of a list you provide
hasmvvalue
Has a missing value
nomvvalue
Does not have a missing value
inornull
Either equals one of a list you provide or is null
notin
Does not equal any of a list you provide
notinornull
Does not equal any of a list you provide or is null
containsoneof
Contains one of a list of substrings you provide
containsnoneof
Does not contain any of the list of substrings you provide
Learn more about filtering expressions generally in this topic: Filtering Expressions.
Use SQL Annotations to Apply Metadata
You can directly specify some column metadata using annotations in your SQL statements, instead of having to separately add metadata via XML. These annotations are case-insensitive and can be used with or without a preceding space, i.e. both "ratio@HIDDEN" and "ratio @hidden" would be valid versions of the first example.@hidden is the same as specifying XML <isHidden>true</isHidden>
SELECT ratio @hidden, log(ratio) as log_ratio
@nolookup is the same as <displayColumnFactory><className>NOLOOKUP</className></displayColumnFactory>. This will show the actual value stored in this field, instead of displaying the lookup column from the target table.
SELECT modifiedby, modifiedby AS userid @nolookup
@title='New Title': Explicitly set the display title/label of the column
SELECT ModifiedBy @title='Previous Editor'
@preservetitle: Don't compute title based on alias, instead keep the title/label of the selected column. (Doesn't do anything for expressions.)
SELECT ModifiedBy as PreviousEditor @preservetitle
Alias Fields
Alias fields are wrapped duplicates of an existing field in a query, to which you can apply independent metadata properties.You can use alias fields to display the same data in alternative ways, or alternative data, such as mapped values or id numbers. Used in conjunction with lookup fields, an alias field lets you display different columns from a target table.
To create an alias field:
Click Alias Field on the Edit Metadata page.
In the popup dialog box, select a field to wrap and click Ok.
A wrapped version of the field will be added to the field list.
You can make changes to relabel or add properties as needed.
Notice that the details column indicates which column is wrapped by this alias.
In cases where the XSD for a given type includes a <sequence> element, the order in which you specify elements matters. If you have elements out of order, you may see an error suggesting that you need instead whatever the "next" expected element could be.Learn more with the example in this topic:
Conditional formats can be specified (1) in the field editor interface (the table definition) and/or (2) as part of a table's metadata XML. When specified in both places, the metadata XML takes precedence over the table definition.In the metadata XML, it is important that the <filters> element is placed first within the <conditionalFormat> element. Modifiers to apply to that filter result, such as bold, italics, and colors must follow the <filters> element.The following adds a yellow background color to any cells showing a "Pulse" value greater than 72.
Be aware that if you don't show a required field to users during insert, they will not actually be able to insert data unless the value is otherwise provided.Two other XML parameters related to hiding columns and making values editable are:
The stored value in your data field may not be how you wish to display that value to users. For example, you might want to show DateTime values as only the "Date" portion, or numbers with a specified number of decimal places. These options can be set on a folder- or site-wide basis, or using the field editor in the user interface as well as within the query metadata as described here.Use a <formatString> in your XML to display a column value in a specific format. They are supported for SQL metadata, dataset import/export and list import/export. The formatString is a template that specifies how to format a value from the column on display output (or on export if the corresponding excelFormatString and tsvFormatString values are not set).Use formats specific to the field type:
Boolean: Values can be formatted using a template of the form positive;negative;null, where "positive" is the string to display when true, "negative" is the string to display when false, and "null" is the string to display when null.
...would make the following formatting changes to fields in a list:
Use a Text Expression for Display
The stored value in your data field may not be how you wish to present that data to users. For example, you might store a "completeness" value but want to show the user a bit more context about that value. Expressions are constructed similar to sample naming patterns using substitution syntax. You can pull in additional values from the row of data as well and use date and number formatting patterns.For example, if you wanted to store a weight value in decimal, but display in sentence form you might something like the following XML for that column:
<column columnName="weight"> <textExpression>${ParticipantID} weighed ${weight} Kg in ${Date:date('MMMM')}.</textExpression> </column>
This example table would look like the first three columns of the following, with the underlying value stored in the weight column only the numeric portion of what is displayed:
ParticipantID
Date
Weight
(Not shown, the stored value for "Weight" column)
PT-111
2021-08-15
PT-111 weighed 69.1 Kg in August.
69.1
PT-107
2021-04-25
PT-107 weighed 63.2 Kg in April.
63.2
PT-108
2020-06-21
PT-108 weighed 81.9 Kg in June.
81.9
Set a Display Width for a Column
If you want to specify a value for the display width of one of your columns (regardless of the display length of the data in the field) use syntax like the following. This will make the Description column display 400 pixels wide:
The default display order of columns in a data entry form is generally based on the underlying order of columns in the database. If you want to present columns in a specific order to users entering data, you can rearrange the user-defined fields using the field editor. If there are built in fields that 'precede' the user-defined ones, you can use the useColumnOrder table attribute. When set to true, columns listed in the XML definition will be presented first, in the order in which they appear in the XML. Any columns not specified in the XML will be listed last in the default order.For example, this XML would customize the entry form for a "Lab Results" dataset to list the ViralLoad and CD4 columns first, followed by the 'standard' study columns like Participant and Timepoint. Additional XML customizations can be applied to the ordered columns, but are not required.
As an alternative to using the UI, you can assign a PHI level to a column in the schema definition XML file. In the example below, the column DeathOrLastContactDate has been marked as "Limited":
A field can use a URL property to open a target URL when the value is clicked. To make this target open in a new tab, use the following syntax to set the target="_blank" attribute. In this example, the link URL has been set using the UI for the "City" column:
Using query metadata provided in a module, you can display images with optional custom thumbnails in a data grid.For example, if you have a store of PNG images and want to display them in a grid, you can put them in a fixed location and refer to them from the grid by URL. The column type in the grid would be a text string, but the grid would display the image. Further, you could provide separate custom thumbnails instead of the miniature version LabKey would generate by default.The module code would use a displayColumnFactory similar to this example, where the properties are defined below.
thumbnailImageUrl(optional) - The URL string expression for the image that will be displayed inline in the grid. If the string expression evaluates to null, the column URL will be used instead. Defaults to the column URL.
thumbnailImageWidth(optional) - The size of the image that will be displayed in the grid. If the property is present but the value is empty (or “native”), no css scaling will be applied to the image. Defaults to “32px”.
thumbnailImageHeight(optional) - The image height
popupImageHeight(optional) - The image height
popupImageUrl(optional) - The URL string expression for the popup image that is displayed on mouse over. If not provided, the thumbnail image URL will be used if the thumbnail image is not displayed at native resolution, otherwise the column URL will be used.
popupImageWidth(optional) - The size of the popup image that will be displayed. If the property is present but the value is empty (or “native”), no css scaling will be applied to the image. Defaults to “300px”.
Using the above example, two images would exist in the 'mystudies' file store: a small blue car image (circled) provided as the thumbnail and the white car image provided as the popup.
Suppress URL and Details Links
Each row in a data grid displays two links to a details page.To suppress these links, add an empty tableURL element <tableUrl/> to the table XML metadata. You can also use this option to suppress linking on a table that looks up into this one.
If you have a query providing calculations for a table, you can add a field alias to the table, then use that to join in your query. In this example, the query "BMI Query" is providing calculations. We added an alias of the "lsid" field, named it "BMI" and then added this metadata XML making that field a foreign key (fk) to the "BMI Query".
Once Physical Exam is linked to BMI Query via this foreign key, all of the columns in BMI Query are made available in the grid customizer. Click Show Hidden Fields to see your wrapped field (or include <isHidden>false</isHidden>), and expand it to show the fields you want.Because these fields are calculated from other data in the system, you will not see them when inserting (or editing) your table.
Premium Resource AvailableSubscribers to premium editions of LabKey Server can learn more with a more detailed example in this topic:
You can then add a field in a table and create a lookup into that query. This can be used both for controlling user input (the user will see a dropdown if you include this field in the insert/update view) as well as for surfacing related columns from the query.
Lookup Display Column
By default, a lookup will display the first non-primary-key text column in the lookup target. This may result in unexpected results, depending on the structure of your lookup target. To control which display column is used, supply the fkDisplayColumnName directly in the XML metadata for the table where you want it displayed:
If you want to filter a lookup column during an insert or update operation, you can do so using XML metadata.For example, consider a list of instruments shared across many departments, each one associated with a specific type of assay (such as elispot) and marked as either in use or not. If your department wants an input form to draw from this shared list, but avoid having an endless menu of irrelevant instruments, you can limit insert choices for an "Instrument" column to only those for the 'elispot' assay that are currently in use by using a filterGroup:
In the above example, the same filter is applied to both insert and update. If you wanted to have different behavior for these two operations, you could separate them like this:
Filter groups like this can be applied to a list, dataset, or any other table that can accept an XML metadata override.
Lookup Sort Order
When you include a lookup, the values presented to the user will be sorted by the lookup's display column by default. If you want to present them to the user in a different order, you can choose another column to use for sorting the values.For example, consider a simple list with these values, in this example it is named "CustomSortedList":
theDisplay
theOrder
a
4
b
2
c
1
d
3
If you have another table looking up into this list for selecting from "theDisplay", users will see a dropdown with "a, b, c, d" listed in that order. If instead you want to present the user "c, b, d, a" in that order, you'd use this XML on the target list (above) in order to force the sorting to be on the "theOrder" column. When you view the list, you will see the rows displayed in this custom order. Note that no custom XML is needed on the table that includes the lookup to this list.
LabKey has basic support for multi-value foreign keys. The multi-valued column renders as a comma separated list in the grid and as a JSON array when using the client API. To set up a multi-value foreign key you need a source table, a junction table, and a target value table. The junction table needs to have a foreign key pointing at the source table and another pointing at the target value table.For example, you can create a Reagent DataClass table as the source, a ReagentSpecies List as the junction table, and a Species List as the lookup target table. Once the tables have been created, edit the metadata XML of the source table to add a new wrapped column over the primary key of the source table and create a lookup that targets the junction table. The wrapped column has a foreign key with these characteristics:
Annotated as multi-valued (<fkMultiValued>junction</fkMultiValued>).
Points to the junction table's column with a foreign key back to the source table (fkColumnName).
Indicates which foreign key of the junction table should be followed to the target value table (fkJunctionLookup).
This example shows the wrappedColumnName "Key", which is the default used for a list. If you are wrapping a column in a DataClass, the wrappedColumnName will be "RowId".
When viewing the multi-value foreign key in the grid, the values will be separated by commas. When using the query APIs such as LABKEY.Query.selectRows, set requiredVersion to 16.2 or greater:
The response format will indicate the lookup is multi-valued in the metaData section and render the values as a JSON array. The example below has been trimmed for brevity:
There are some limitations of the multi-value foreign keys, however. Inserts, updates, and deletes must be performed on the junction table separately from the insert into either the source or target table. In future versions of LabKey Server, it may be possible to perform an insert on the source table with a list of values and have the junction table populated for you. Another limitation is that lookups on the target value are not supported. For example, if the Author had a lookup column named "Nationality", you will not be able to traverse the Book.Author.Nationality lookup.
Disables the standard insert, update, and delete buttons/links with the empty <insertUrl /> and other tags.
Configures lookups on a couple of columns and hides the RowId column in some views.
Adds a custom button "More Actions" with a child menu item "Limit To Animals In Selection" that calls a JavaScript method provided in a referenced .js file.
Adds a button to the Sample Types web part. When the user selects samples and clicks the button, the page performCallSort.html is shown, where the user can review the selected records before exporting the records to an Excel file.
To use this sample, place Aliquots.query.xml in a module's ./resources/queries/Samples directory. Rename Aliquots.query.xml it to match your sample type's name. Edit the tableName attribute in the Aliquots.query.xml to match your sample type's name. Replace the MODULE_NAME placeholder with the name of your module. Place the HTML file in your module's ./resources/views directory. Edit the queryName config parameter to match the sample type's name.
Custom SQL queries give you the ability to flexibly present the data in a table in any way you wish. To edit a query's name, description, or visibility properties, follow the instructions in this topic.
Select the schema and query/table of interest and then click Edit Properties.
The Edit query properties page will appear, for example:
Available Query Properties
Name: This value appears as the title of the query/table in the Schema Browser, data grids, etc.Description: Text entered here will be shown with your query in the Schema Browser.Available in child folders: Queries live in a particular folder on LabKey Server, but can be marked as inheritable by setting this property to 'yes'. Note that the query will only be available in child folders containing the matching base schema and table.Hidden from the user: If you set this field to 'yes', then the query will no longer appear in the Schema Browser, or other lists of available queries. It will still appear on the list of available base queries when you create a new query.
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Query dependencies are traced and noted in the schema browser in a Dependency Report, below the lists of columns and indices. Particularly for user-defined queries derived from built in queries and tables it can be helpful to track the sources and dependents when you make any query changes.
To see the Dependency Report, select > Go To Module > Query and select the schema and table or query of interest.Shown below, the "AverageTemp + Physical Exam" query depends on both the table "Physical Exam" and another query, "AverageTempPerParticipant". Notice the different icon indicators. In turn, this query was used to base another query "Physical Exam + TempDelta" shown under Dependents.If there are no dependencies for the query, this section will not be shown.
Note that when the dependency report is run, it will need to determine the columns in each query. For a PIVOT query, this may have performance implications, causing it to appear as if the dependency report does not load. Learn about possible optimizations in this topic: Pivot Queries
Lookups in Dependency Reports
The Dependency Report also includes lookups between tables and queries. For example, if in a study, the "Demographics" dataset has a lookup into the "Languages" list, this relationship will be included in both Dependency Reports:
Trace Cross-Folder Query Dependencies
Select > Go To Module > Query.
Click Cross Folder Dependencies.
Select the scope of the analysis:
Site Level: Perform analysis across all folders and projects on the server.
Current Project Level: Perform analysis within the current project and all subfolders of it.
Click Start Analysis.
Once the analysis begins, the entire schema browser will be masked and a progress indicator will be shown. Folder names will be displayed as they are analyzed. When the analysis is complete, a message box will appear. Click OK to close.All query detail tabs that are currently open will have their dependency report refreshed to display any information that was added from the cross folder analysis.
The Query web part can be created by a page administrator and used to display specific information of use to users, in a flexible interface panel that can show either:
A list of all queries in a particular schema
A specific query or grid view
Add a Query Web Part
Navigate to where you want to display the query.
Enter > Page Admin Mode.
Click Select Web Part drop-down menu at the bottom left of the page, select “Query” and click Add.
Provide the necessary information:
Web Part Title: Enter the title for the web part, which need not match the query name.
Schema: Pull down to select from available schema.
View: By default, the default grid will be shown; pull down if you want to select a custom grid view.
Allow user to choose query?: If you select "Yes", the web part will allow the user to change which query is displayed. Only queries the user has permission to see will be available.
Allow user to choose view?: If you select "Yes", the web part will allow the user to change which grid view is displayed. Only grid views the user has permission to see will be available.
Button bar position: Select whether to display web part buttons at the top of the grid. Options: Top, None.
Once you've customized the options:
Click Submit.
Click Exit Admin Mode.
List of All Queries
When you select the list of queries in the schema, the web part will provide a clickable list of all queries the user has access to view.
Show Specific Query and View
The web part for a specific query and view will show the grid in a web part similar to the following. If you selected the option to allow the user to select other queries (or views) you will see additional menus:
This topic covers using JOIN in LabKey SQL to bring data together. All data is represented in tables, or queries. A list or dataset or user defined query are all examples of queries. Queries are always defined in a specific folder (container) and schema, but can access information from other queries, schema, and folders using JOINs.
Use JOIN to combine columns from different tables.The following query combines columns from the Physical Exam and Demographics tables.
SELECT PhysicalExam.ParticipantId, PhysicalExam.weight_kg, Demographics.Gender, Demographics.Height FROM PhysicalExam INNER JOIN Demographics ON PhysicalExam.ParticipantId = Demographics.ParticipantId
Note that when working with datasets in a study, every dataset also has an implied "Datasets" column that can be used in select statements instead of join. See an example in this topic: More LabKey SQL Examples
Handle Query/Table Names with Spaces
When a table or query name has spaces in it, use "double quotes" around it in your query. For example, if the table name "Physical Exam" had a space in it, the above example would look like:
SELECT "Physical Exam".ParticipantId, "Physical Exam".weight_kg, Demographics.Gender, Demographics.Height FROM "Physical Exam" INNER JOIN Demographics ON "Physical Exam".ParticipantId = Demographics.ParticipantId
Handle Duplicate Column Names
When joining two tables that have some column names in common, the duplicates will be disambiguated by appending "_1", "_2" to the joined column names as needed. The first time the column name is seen in the results, no number is appended.See an example here: More LabKey SQL Examples
Combine Tables with FULL OUTER JOIN
The following example shows how to join all rows from two datasets (ViralLoad and ImmuneScore), so that the columns RNACopies and Measure1 can be compared or visualized. In this example, we handle the duplicate column names (ParticipantId and VisitID) using CASE/WHEN statements, though given the join is always on these two columns matching, we could also just select from either table directly.
SELECT CASE WHEN ViralLoad.ParticipantId IS NULL THEN ImmuneScore.ParticipantId WHEN ViralLoad.ParticipantId IS NOT NULL THEN ViralLoad.ParticipantId END AS ParticipantId, CASE WHEN ViralLoad.VisitID IS NULL THEN ImmuneScore.VisitID WHEN ViralLoad.VisitID IS NOT NULL THEN ViralLoad.VisitID END AS VisitID, ViralLoad.RNACopies, ImmuneScore.Measure1, FROM ViralLoad FULL OUTER JOIN ImmuneScore ON ViralLoad.ParticipantId = ImmuneScore.ParticipantId AND ViralLoad.VisitID = ImmuneScore.VisitID
CROSS JOIN
Cross joins give you all possible combinations between two different tables (or queries).For example, suppose you have two tables Things and Items:
Thing
A
B
Item
1
2
Then cross join will provide the set of all possible combinations between the two tables:
Thing
Item
A
1
A
2
B
1
B
2
SELECT Things.Name AS Thing, Items.Name AS Item FROM Items CROSS JOIN Things
JOIN Queries Across Schema
You can join tables from two different schemas in the same data source by using the syntax "JOIN <SCHEMA>.<TABLE>" to refer to the table in another schema.
Note: you cannot create joins across data sources. To join native tables to data from an external schema, you must first ETL the external data into the local database. Learn more here:
For example, suppose you have the following tables in your local database, one in the study schema, the other in the lists schema:study.Demographics
ParticipantId
Language
Gender
PT-101
English
M
PT-102
French
F
PT-103
German
F
lists.Languages
LanguageName
TranslatorName
TranslatorPhone
English
Monica Payson
(206) 234-4321
French
Etiene Anson
206 898-4444
German
Gundula Prokst
(444) 333-5555
The following SQL JOIN creates a table that combines the data from both source tables.
SELECT Demographics.ParticipantId, Demographics.Language, Languages.TranslatorName, Languages.TranslatorPhone, Demographics.Gender FROM Demographics JOIN lists.Languages ON Demographics.Language=Languages.LanguageName
The data returned:
ParticipantId
Language
TranslatorName
TranslatorPhone
Gender
PT-101
English
Monica Payson
(206) 234-4321
M
PT-102
French
Etiene Anson
206 898-4444
F
PT-103
German
Gundula Prokst
(444) 333-5555
F
JOIN Queries Across Folders
You can join queries from two different containers by adding the server path to the syntax in your JOIN statement. For example, if you had a project named "Other" containing a folder named "Folder" you could use a list in it from another folder by following this example:
SELECT Demographics.ParticipantId, Demographics.Language, Languages.TranslatorName, Languages."PhoneNumber", FROM Demographics JOIN "/Other/Folder".lists.Languages ON Demographics.Language=Languages.Language
Learn more about queries across folders in this topic:
In this example we use SQL to add a column to a query based on the Physical Exam dataset. The column will display "Pulse Pressure" -- the change in blood pressure between contractions of the heart muscle, calculated as the difference between systolic and diastolic blood pressures.
Create a Query
Navigate to > Go To Module > Query.
Select a schema to base the query on. (For this example, select the study schema.)
Click Create New Query.
Create a query with the name of your choice, based on some dataset. (For this example, select the Physical Exam dataset.)
Click Create and Edit Source.
Modify the SQL Source
The default query provided simply selects all the columns from the chosen dataset. Adding the following SQL will create a column with the calculated value we seek. Note that if your query name included a space, you would put double quotes around it, like "Physical Exam". Subtract the diastolic from the systolic to calculate pulse pressure:
PhysicalExam.systolicBP-PhysicalExam.diastolicBP as PulsePressure
The final SQL source should look like the following, with the new line added at the bottom. Remember to add a comma on the previous line:
SELECT PhysicalExam.ParticipantId, PhysicalExam.date, PhysicalExam.weight_kg, PhysicalExam.temperature_C, PhysicalExam.systolicBP, PhysicalExam.diastolicBP, PhysicalExam.pulse, PhysicalExam.systolicBP-PhysicalExam.diastolicBP as PulsePressure FROM PhysicalExam
Click Save and Finish.
Notice that LabKey Server has made a best guess at the correct column label, adding a space to "Pulse Pressure".
To reopen and further edit your query, return to the schema browser, either using the study Schema link near the top of the page or the menu. Your new query will be listed in the study schema alphabetically.
Example #2: BMI
The following query pair shows how to use intermediate and related queries to calculate the Body Mass Index (BMI). In this scenario, we have a study where height is stored in a demographic dataset (does not change over time) and weight is recorded in a physical exam dataset. Before adding this BMI calculation, we create an intermediate query "Height and Weight" using the study "Datasets" table alignment to provide weight in kg and height in meters for our calculation.Height and Weight query:
SELECT PhysicalExam.ParticipantId, PhysicalExam.date, PhysicalExam.weight_kg, TRUNCATE(CAST(Datasets.Demographics.height AS DECIMAL)/100, 2) as height_m, FROM PhysicalExam
BMI query:
SELECT "Height and Weight".ParticipantId, "Height and Weight".Date, "Height and Weight".weight_kg, "Height and Weight".height_m, ROUND(weight_kg / (height_m * height_m), 2) AS BMI FROM "Height and Weight"
Query Key Fields
If you intend to use a custom query in a join, note that you may need to explicitly specify a primary key appropriate to the join. Learn more here: Query Metadata: Examples
Display the Calculated Column in the Original Table
Premium Resource AvailableSubscribers to premium editions of LabKey Server can learn how to display a calculated column from a query in the original table following this topic:
A pivot query helps you summarize and re-visualize data in a table. Data can be grouped or aggregated to help you focus on a particular aspect of your data. For example a pivot table can help you see how many data points of a particular kind are present, or it can represent your data by aggregating it into different categories.
To try this yourself, you can download this sample file and import it as a Standard assay type. Remember your assay design name; we use "PivotDemo" below.
Select a schema. In our example, we used the assay schema, specifically "assay.General.PivotDemo" in our example.
Click Create New Query.
Name it and confirm the correct query/table is selected. For our assay, we chose "Data" (the assay results).
Click Create and Edit Source.
Syntax for PIVOT...BY Query
A PIVOT query is essentially a SELECT specifying which columns you want and how to PIVOT and GROUP them. To write a pivot query, follow these steps.(1) Start with a base SELECT query.
SELECT ParticipantID, Date, RunName, M1 FROM Data
(2) Identify the data cells you want to pivot and how. In this example, we focus on the values in the RunName column, to separate M1 values for each.(3) Select an aggregating function to handle any non-unique data, even if you do not expect to need it. MAX, MIN, AVG, and SUM are possibilities. Here we have only one row for each participant/date/run combination, so all would produce the same result, but in other data you might have several sets with multiple values. When aggregating, we can also give the column a new name, here MaxM1.(4) Identify columns which remain the same to determine the GROUP BY clause.
SELECT ParticipantId, Date, RunName, Max(M1) as MaxM1 FROM Data GROUP BY ParticipantId, Date, RunName
(5) Finally, pivot the cells.
SELECT ParticipantId, Date, RunName, Max(M1) as MaxM1 FROM Data GROUP BY ParticipantId, Date, RunName PIVOT MaxM1 by RunName
PIVOT...BY...IN Syntax
You can focus on particular values using an IN clause to identify specific columns (i.e. distinct rows in the original data) to use in the pivot.(6) In our example from above, perhaps we want to see only data from two of the runs:
SELECT ParticipantId, Date, RunName, Max(M1) as MaxM1 FROM Data GROUP BY ParticipantId, Date, RunName PIVOT MaxM1 by RunName IN ('Run1', 'Run2')
Note that pivot column names are case-sensitive. You may need to use lower() or upper() in your query to work around this issue if you have two values who differ only by letter case.In addition to reducing the set of result columns, specifying column names in an IN clause has a performance benefit, since LabKey won't need to execute the query to determine the column set.
PIVOT...IN using Sub-SELECT
If you won't know the specific column names for an IN clause (i.e. the distinct row values in the original data) at the time of writing your query, you can also use a sub-SELECT statement in the IN clause. You could either pull the column names from a separate list, a simpler query, or as in this simple query, from values in the same table you will pivot. For example:
SELECT Language, Gender, MAX(Height) AS H FROM Demographics GROUP BY Language, Gender PIVOT H By Gender IN (SELECT DISTINCT Gender FROM Demographics)
Grouped Headers
You can add additional aggregators and present two pivoted columns, here we show both an minimum and a maximum M1 value for each participant/date/run combination. In this case they are the same, but the pattern can be applied to varying data.
SELECT ParticipantId, Date, RunName, Min(M1) as MinM1, Max(M1) as MaxM1 FROM Data GROUP BY ParticipantId, Date, RunName PIVOT MinM1, MaxM1 by RunName IN ('Run1', 'Run2')
"Summary" Columns
"Summary" columns are those columns not included in the group-by or pivoted list. When generating the pivot, these summary columns are aggregated as follows:
a COUNT or SUM aggregate summary column is wrapped with SUM
a MIN or MAX is wrapped with a MIN or MAX
For example:
SELECT -- group by columns AssignedTo, Type,
-- summary columns. Turned into a SUM over the COUNT(*) COUNT(*) AS Total,
-- pivoted columns SUM(CASE WHEN Status = 'open' THEN 1 ELSE 0 END) AS "Open", SUM(CASE WHEN Status = 'resolved' THEN 1 ELSE 0 END) AS Resolved
FROM issues.Issues WHERE Status != 'closed' GROUP BY AssignedTo, Type PIVOT "Open", Resolved BY Type IN ('Defect', 'Performance', 'To Do')
Options for Pivoting by Two Columns
Two levels of PIVOT are not supported, however some options for structuring your query may give you the behavior you need. In this scenario, you have experiment data and want to pivot by both peak intensity (not known up front) and sample condition.The closest option is to concatenate the two values together and pivot on that "calculated" column.
SELECT Run.Batch, Run.Sample, COUNT(*) AS ResultsCount, Run.SampleCondition || ' ' || PeakLabel AS ConditionPeak, AVG(Data.PercTimeCorrArea) AS AvgPercTimeCorrArea, FROM Data GROUP BY Run.Batch, Run.Sample, Run.SampleCondition || ' ' || PeakLabel PIVOT AvgPercTimeCorrArea BY ConditionPeak
If concatenation is not practical, you could use a version like this. It will include duplicate Batch and Sample columns, which you could get rid of by explicitly listing the aggregates from each sample condition sub-query. Syntax like "SELECT NR.* EXCEPT (NR_Batch, NR_Sample)" to get all of the pivot columns except the joining columns is not supported.
SELECT NR.*, RED.* FROM (
SELECT Run.Batch AS NR_Batch, Run.Sample AS NR_Sample, --Run.SampleCondition, PeakLabel, AVG(Data.PercTimeCorrArea) AS NR_AvgPercTimeCorrArea, FROM Data WHERE Run.SampleCondition = 'NR' GROUP BY Run.Batch, Run.Sample, Run.SampleCondition, PeakLabel PIVOT NR_AvgPercTimeCorrArea BY PeakLabel
) NR
INNER JOIN (
SELECT Run.Batch AS RED_Batch, Run.Sample AS RED_Sample, --Run.SampleCondition, PeakLabel, AVG(Data.PercTimeCorrArea) AS RED_AvgPercTimeCorrArea, FROM Data WHERE Run.SampleCondition = 'RED' GROUP BY Run.Batch, Run.Sample, Run.SampleCondition, PeakLabel PIVOT RED_AvgPercTimeCorrArea BY PeakLabel
) RED ON NR.NR_Batch = RED.RED_Batch AND NR.NR_Sample = RED.RED_Sample
Examples
Example 1 - Open issues by priority for each area
Another practical application of a pivot query is to display a list of how many issues of each priority are open for each feature area.
SELECT Issues.Area, Issues.Priority, Count(Issues.IssueId) AS CountOfIssues FROM Issues GROUP BY Issues.Area, Issues.Priority PIVOT CountOfIssues BY Priority IN (1,2,3,4)
Example 2 - Open issues by type grouped by who they are assigned to
SELECT -- Group By columns AssignedTo, Type,
-- Summary columns. Turned into a SUM over the COUNT(*) COUNT(*) AS Total,
-- Pivoted columns SUM(CASE WHEN Status = 'open' THEN 1 ELSE 0 END) AS "Open", SUM(CASE WHEN Status = 'resolved' THEN 1 ELSE 0 END) AS Resolved
FROM issues.Issues WHERE Status != 'closed' GROUP BY AssignedTo, Type PIVOT "Open", Resolved BY Type IN ('Defect', 'Performance', 'To Do')
Example 3 - Luminex assay fluorescence intensity by analyte
SELECT ParticipantId, date, Analyte, Count(Analyte) AS NumberOfValues, AVG(FI) AS AverageFI, MAX(FI) AS MaxFI FROM "Luminex Assay 100" GROUP BY ParticipantID, date, Analyte PIVOT AverageFI, MaxFI BY Analyte
Example 4 - Support tickets by priority and status
SELECT SupportTickets.Client AS Client, SupportTickets.Status AS Status, COUNT(CASE WHEN SupportTickets.Priority = 1 THEN SupportTickets.Status END) AS Pri1, COUNT(CASE WHEN SupportTickets.Priority = 2 THEN SupportTickets.Status END) AS Pri2, COUNT(CASE WHEN SupportTickets.Priority = 3 THEN SupportTickets.Status END) AS Pri3, COUNT(CASE WHEN SupportTickets.Priority = 4 THEN SupportTickets.Status END) AS Pri4, FROM SupportTickets WHERE SupportTickets.Created >= (curdate() - 7) GROUP BY SupportTickets.Client, SupportTickets.Status PIVOT Pri1, Pri2, Pri3, Pri4 BY Status
You can perform cross-folder queries by identifying the folder that contains the data of interest during specification of the dataset. The path of the dataset is composed of the following components, strung together with a period between each item:
Project - This is literally the word Project, which is a virtual schema and resolves to the current folder's project.
Path to the folder containing the dataset, surrounded by quotes. This path is relative to the current project. So a dataset located in the Home > Tutorials > Demo subfolder would be referenced using "Tutorials/Demo/".
Schema name - In the example below, this is study
Dataset name - Surrounded by quotes if there are spaces in the name. In the example below, this is "Physical Exam"
Note that LabKey Server enforces a user's security roles when they view a cross-folder query. To view a cross folder/container query, the user must have the Reader role in each of the component folders/containers which the query draws from.ExampleThe Edit SQL Query Source topic includes a simple query on the "Physical Exam" dataset which looks like this, when the dataset is in the local folder:
SELECT "Physical Exam".ParticipantID, ROUND(AVG("Physical Exam".Temp_C), 1) AS AverageTemp FROM "Physical Exam" GROUP BY "Physical Exam".ParticipantID
This query could reference the same dataset from a sibling folder within the same project. To do so, you would replace the string used to identify the dataset in the FROM clause ("Physical Exam" in the query used in this topic) with a fully-specified path. For this dataset, you would use:
Project."Tutorials/Demo/".study."Physical Exam"
Using the query from the example topic as a base, the cross-folder verson would read:
SELECT "Physical Exam".ParticipantID, ROUND(AVG("Physical Exam".Temp_C), 1) AS AverageTemp FROM Project."Tutorials/Demo/".study."Physical Exam" GROUP BY "Physical Exam".ParticipantID
Cross-Project Queries
You can perform cross-project queries using the full path for the project and folders that contain the dataset of interest. To indicate that a query is going across projects, use a full path, starting with a slash. The syntax is "/<FULL FOLDER PATH>".<SCHEMA>.<QUERY>
Full path to the folder containing the dataset, surrounded by quotes. This lets you access an arbitrary folder, not just a folder in the current project. You can use a leading / slash in the quoted path to indicate starting at the site level. You can also use the virtual schema "Site." to lead your path, as shown below in the module property example. For example, a dataset located in the Tutorials > Demo subfolder would be referenced from another project using "/Tutorials/Demo/".
Schema name - In the example below, this is study
Dataset name - Surrounded by quotes if there are spaces in the name. In the example below, this is "Physical Exam"
Example 1The example shown above can be rewritten using cross-project syntax by including the entire path to the dataset of interest in the FROM clause, preceded by a slash.
"/Tutorials/Demo/".study."Physical Exam"
Using the query from the example topic as a base, the cross-folder verson would read:
SELECT "Physical Exam".ParticipantID, ROUND(AVG("Physical Exam".Temp_C), 1) AS AverageTemp FROM "/Tutorials/Demo/".study."Physical Exam" GROUP BY "Physical Exam".ParticipantID
Example 2
SELECT Demographics.ParticipantId, Demographics.Language, Languages.TranslatorName, Languages."PhoneNumber", FROM Demographics JOIN "/Other/Folder".lists.Languages ON Demographics.Language=Languages.Language
Use ContainerFilter in SQL
You can annotate individual tables in the FROM clause with an optional container filter if you want to restrict or expand the scope of the individual table in a larger query. For example, this would allow an issues report to specify that the query should use CurrentAndSubfolder without having to create a corresponding custom view. Further, the defined container filter can not be overridden by a custom view.The syntax for this is:
SELECT * FROM Issues [ContainerFilter='CurrentAndSubfolders']
Find the list of possible values for the containerFilter at this link. In SQL the value must be surrounded by single quotes and capitalized exactly as expected, i.e. with a leading capital letter.
This option can be used in combination with other SQL syntax, including the cross-folder and cross-project options above.
Use Module Properties
You can specify the path for a cross-folder or cross-project query using a module property, allowing you to define a query to apply to a different path or dataset target in each container. For example, if you define the module property "DatasetPathModuleProp" in the module "MyCustomModule", the syntax to use would be:
SELECT source.ParticipantId FROM Site.{moduleProperty('MyCustomModule', 'DatasetPathModuleProp')}.study.source
In each container, you would define this module property to be the full path to the desired location.
Rules about interpreting schemas and paths:"Site" and "Project" are virtual schemas that let you resolve folder names in the same way as schema names. It is inferred that no schema will start with /, so if a string starts with / it is assumed to mean the site folder and the path will be resolved from there.While resolving a path, if we have a folder schema in hand, and the next schema contains "/", it will be split and treated as a path.
Fields with Dependencies
A few LabKey fields/columns have dependencies. To use a field with dependencies in a custom SQL query, you must explicitly include supporting fields.To use Assay ID in a query, you must include the run's RowId and Protocol columns. You must also use these exact names for the dependent fields. RowId and Protocol provide the Assay ID column with data for building its URL.If you do not include the RowId and Protocol columns, you will see an error for the Run Assay ID field. The error looks something like this:"KF-07-15: Error: no protocol or run found in result set."
LabKey Server lets you add parameters to your SQL queries, using the PARAMETERS keyword. When a user views the query, they will first assign values to the parameters (seeing any default you provide) and then execute the query using those values.
Example Parameterized SQL Query
The following SQL query defines two parameters, MinTemp and MinWeight:
SELECT PhysicalExam.ParticipantId, PhysicalExam.date, PhysicalExam.weight_kg, PhysicalExam.temperature_C FROM PhysicalExam WHERE temperature_C >= MinTemp and weight_kg >= MinWeight
By default, parameterized queries are hidden in the Schema Browser. Select Show Hidden Schemas and Queries to view. Go to the Schema Browser, and look in the far lower left. For details, see SQL Query Browser.
Note: Only constants are supported for default parameter values.
Example Using the URL to Pass Parameter Values
To pass a parameter value to the query on the URL use the syntax:
The parameters are written into the request URL as follows:
query.param.MinTemp=36&query.param.MinWeight=90
User Interface for Parameterized SQL Queries
You can also pass in values using a built-in user interface. When you view a parameterized query in LabKey Server, a form is automatically generated, where you can enter values for each parameter.
Go to the Schema Browser: > Go To Module > Query.
On the lower left corner, select Show Hidden Schemas and Queries. (Parameterized queries are hidden by default.)
Locate and select the parameterized query.
Click View Data.
You will be presented with a form, where you can enter values for the parameters:
ETLs and Parameterized SQL Queries
You can also use parameterized SQL queries as the source queries for ETLs. Pass parameter values from the ETL into the source query from inside the ETL's config XML file. For details see ETL: Examples.
When joining two tables that have some column names in common, the duplicates will be disambiguated by appending "_1", "_2" to the joined column names as needed. The first time the column name is seen in the results, no number is appended.For example, the results for this query would include columns named "EntityId", etc. from c1, and "EntityId_1", etc. from c2:
SELECT * FROM core.Containers c1 INNER JOIN core.Containers c2 ON c1.parent = c2.entityid
Note that the numbers are appended to the field key, not the caption of a column. The user interface displays the caption, and thus may omit the underscore and number. If you hover over the column, you will see the field key, and if you export data with "Column headers=Field key" the numbered column names will be included.If you want to be able to display to users which field came from which table, you can edit the originating table definition to add either a distinct field label, or a Description including the origin of the field, which will be shown to users on hover.
"Datasets" Special Column in Studies
SQL queries based on study datasets can make use of the special column "Datasets", which gives access to all datasets in the current study.For example, the following query on the PhysicalExam dataset uses the "Datasets" special column to pull in data from the Demographics dataset. You can use this query in the tutorial example study.
SELECT PhysicalExam.ParticipantId, PhysicalExam.Weight_kg, PhysicalExam.Datasets.Demographics.Height, PhysicalExam.Datasets.Demographics.Country FROM PhysicalExam
"Datasets" provides a shortcut for queries that would otherwise use a JOIN to pull in data from another dataset. The query above can be used instead of the JOIN style query below:
SELECT PhysicalExam.ParticipantId, PhysicalExam.Weight_kg, Demographics.Height, Demographics.Country FROM PhysicalExam JOIN Demographics ON PhysicalExam.ParticipantId = Demographics.ParticipantId
Use "GROUP BY" for Calculations
The GROUP BY function is useful when you wish to perform a calculation on a table that contains many types of items, but keep the calculations separate for each type of item. You can use GROUP BY to perform an average such that only rows that are marked as the same type are grouped together for the average.For example, what if you wish to determine an average for each participant in a large study dataset that spans many participants and many visits. Simply averaging a column of interest across the entire dataset would produce a mean for all participants at once, not each participant. Using GROUP BY allows you to determine a mean for each participant individually.
GROUP BY Example: Average Temp Per Participant
The GROUP BY function can be used on the PhysicalExam dataset to determine the average temperature for each participant across all of his/her visits.To set up this query, follow the basic steps described in the Create a SQL Query example to create a new query based on the PhysicalExam table in the study schema. Name this new query "AverageTempPerParticipant."
If you are working with the LabKey tutorial study, these queries may be predefined, so you can view and edit them in place, or create new queries with different names.
Within the SQL Source editor, delete the SQL created there by default for this query and paste in the following SQL:
SELECT PhysicalExam.ParticipantID, ROUND(AVG(PhysicalExam.temperature_c), 1) AS AverageTemp, FROM PhysicalExam GROUP BY PhysicalExam.ParticipantID
For each ParticipantID, this query finds all rows for that ParticipantID and calculates the average temperature for these rows, rounded up to the 10ths digit. In other words, we calculate the participant's average temperature across all visits and store that value in a new column called "AverageTemp."See also the Efficient Use of "GROUP BY" section below.
JOIN a Calculated Column to Another Query
The JOIN function can be used to combine data in multiple queries. In our example, we can use JOIN to append our newly-calculated, per-participant averages to the PhysicalExam dataset and create a new, combined query.First, create a new query based on the "PhysicalExam" table in the study schema. Call this query "PhysicalExam + AverageTemp" and edit the SQL to look like:
SELECT PhysicalExam.ParticipantId, PhysicalExam.date, PhysicalExam.weight_kg, PhysicalExam.temperature_C, PhysicalExam.systolicBP, PhysicalExam.diastolicBP, PhysicalExam.pulse, AverageTempPerParticipant.AverageTemp FROM PhysicalExam INNER JOIN AverageTempPerParticipant ON PhysicalExam.ParticipantID=AverageTempPerParticipant.ParticipantID
You have added one line before the FROM clause to add the AverageTemp column from the AverageTempPerParticipant dataset. You have also added one additional line after the FROM clause to explain how data in the AverageTempPerParticipant are mapped to columns in the PhysicalExam table. The ParticipantID column is used for mapping between the tables.
Premium Resource AvailableSubscribers to premium editions of LabKey Server can learn about displaying a calculated column in the original table following the example in this topic:
We next use our calculated columns as the basis for creating yet another calculated column that provides greater insight into our dataset.This column will be the difference between a participant's temperature at a particular visit and the average temperature for all of his/her visits. This "TempDelta" statistic will let us look at deviations from the mean and identify outlier visits for further investigation.Steps:
Create a new query named "PhysicalExam + TempDelta" and base it on the "PhysicalExam + AverageTemp" query we just created above.
Filter Calculated Column to Make Outliers Stand Out
It can be handy to filter your results such that outlying values stand out. This is simple to do in a LabKey grid using the column header filtering options.Using the query above ("PhysicalExam + TempDelta"), we want to show the visits in which a participant's temperature was unusually high for them, possibly indicating a fever. We filter the calculated "Temperature Difference From Average" column for all values greater than 1.5. Just click on the column header, select Filter. Choose "Is Greater Than" and type "1.0" in the popup, then click OK.This leaves us with a list of all visits where a participant's temperature was more than 1 degree C above the participant's mean temperature at all his/her visits. Notice the total number of filtered records is displayed above the grid.
Use "SIMILAR_TO"
String pattern matching can be done using similar_to. The syntax is similar_to(A,B,C): A similar to B escape C. The escape clause is optional.
'A' is the string (or field name) to compare
'B' is the pattern to match against
'C' is the escape character (typically a backslash) used before characters that would otherwise be parsed as statement syntax, including but not limited to "%", "(", ",".
To return all the names on a list that started with AB, you might use:
SELECT Name from MyList WHERE similar_to (Name, 'AB%')
If you wanted to return names on the list that started with '%B', you would use the following which uses a to escape the first % in the string.:
If you have data where there might be an arbitrary number of spaces between terms that you want to match in a pattern string, use SIMILAR_TO with a space followed by + to match an unknown number of spaces. For example, suppose you wanted to find entries in a list like this, but didn't know how many spaces were between the term and the open parentheses "(".
Thing (one)
Thing (two - with two spaces)
Thing to ignore
Thing (three)
Your pattern would need to both include " +" to match one or more spaces and escape the "(":
WHERE similar_to(value, 'Thing +\(%', '\')
Round a Number and Cast to Text
If your query involves both rounding a numeric value (such as to 1 or two digits following the decimal place) and also casting that value to text, the CAST to VARCHAR will "undo" the rounding adding extra digits to your text value. Instead, use to_char. Where 'myNumber' is a numeric value, pass the formatting to use as the second parameter. to_char will also round the value to the specified number of digits:
In some use cases, it can seem logical to have a long GROUP BY list. This can be problematic when the tables are large, and there may be a better way to write such queries both for performance and for readability. As a general rule, keeping GROUP BY clauses short is best practice for more efficient queries.As an example, imagine you have customer and order tables. You want to get all of the customer's info plus the date for their most recent order.You could write something like:
SELECT c.FirstName, c.LastName, c.Address1, c.Address2, c.City, c.State, c.Zip, MAX(o.Date) AS MostRecentOrder FROM Customer c LEFT JOIN Order o ON c.Id = o.CustomerId GROUP BY c.FirstName, c.LastName, c.Address1, c.Address2, c.City, c.State, c.Zip
...but that makes the database do the whole join (which may multiply the total number of rows) and then sort it by all of the individual columns so that it can de-duplicate them again for the GROUP BY.Instead, you could get the most recent order date via a correlated subquery, like:
SELECT c.FirstName, c.LastName, c.Address1, c.Address2, c.City, c.State, c.Zip, (SELECT MAX(o.Date) FROM Order o WHERE c.Id = o.CustomerId) AS MostRecentOrder FROM Customer c
Or, if you want more than one value from the Order table, change the JOIN so that it does the aggregate work:
SELECT c.FirstName, c.LastName, c.Address1, c.Address2, c.City, c.State, c.Zip, o.MostRecentOrder, o.FirstOrder FROM Customer c LEFT JOIN (SELECT CustomerId, MAX(Date) AS MostRecentOrder), MIN(Date) AS FirstOrder FROM Order GROUP BY CustomerId) o ON c.Id = o.CustomerId
"COALESCE" Same-named Columns
If you have multiple data structures that are similar in having some of the "same" fields, but stored in distinct structures, you cannot directly join these fields in a query, but provided they are of the same type, you can use COALESCE to 'fold' them together.For example, say you have two Sample Types, "Blood" and "Plasma" and each has a "MAIN_ID" field and a "Date" field. You want to print a grid to be used to make an inventory list for a mixed box of samples. You can add samples of both types to a list (keyed on the "Name" field of each sample), such as a Picklist in Sample Manager or Biologics. Any multi-Sample Type grid will include fields common to all Sample Types on one tab, but this is limited to built in fields that the system has internally aligned. In this example, we're adding additional fields but the system does not have a way of knowing that they are "related".You could print a multi-sheet Excel file from a picklist, separating the fields for the two types from within the application. In order to create a "combined" grid showing the corresponding values of these Sample Type-specific fields in one place, you could use "COALESCE" to combine the different table columns into one:
SELECT PrintThisBox.SampleID, PrintThisBox.SampleId.SampleSet.Name As "Sample Type", COALESCE(Blood.MAIN_ID, Plasma.MAIN_ID, 'None') AS MAIN_ID, COALESCE(Blood.Date, Plasma.Date) AS "Sample Date" FROM PrintThisBox LEFT JOIN samples.Blood on PrintThisBox.SampleId.Name = Blood.Name LEFT JOIN samples.Plasma on PrintThisBox.SampleId.Name = Plasma.Name
In this example, we're making use of a picklist named "PrintThisBox" containing samples of the two types. The "MAIN_ID" field in each Sample Type will be COALESCED into a "blended" MAIN_ID column, with any rows failing to have values for it shown as 'None'. In our example, we know that the "Date" field is always populated, so don't need to provide a default value. As many tables as needed could be "folded" into a shared set of columns in this manner, as COALESCE will look in each row for the first one of the provided arguments that exists.
Linked schemas allow you to access data from another folder, by linking to a schema in that folder. Linked schemas are useful for providing read access to selected data in another folder which is otherwise not allowed by that folder's overall permissions. Linked schemas provide a way to apply security settings at a finer granularity than at the level of whole folders.
Linked schemas are especially useful when you want to reveal some data in a folder without granting access to the whole folder. For example, suppose you have the following data in a single folder A. Some data is intended to be private, some is intended for the general public, and some is intended for specific audiences:
Private data to be shown only to members of your team.
Client data and tailored views to be shown to individual clients.
Public data to be shown on a portal page.
You want to reveal each part of the data as is appropriate for each audience, but you don't want to give out access to folder A. To do this, you can create a linked schema in another folder B that exposes selected items in folder A. The linked schema may expose some or all of the tables and queries from the original schema. Furthermore, the linked tables and queries may be additionally filtered to create more refined views, tailored for specific audiences.
Security Considerations
The nature of a linked schema is to provide access to tables and queries in another folder which likely has different permissions than the current folder.Permissions: A linked schema always grants the standard "Reader" role in the target folder to every user who can access the schema. Site Administrators who configure a linked schema must be extremely careful, since they are effectively bypassing folder and dataset security for the tables/queries that are linked. In addition, since the "Reader" role is granted in only the target folder itself, a linked schema can't be used to execute a "cross-container" query, a custom query that references other folders. If multiple containers must be accessed then multiple linked schemas will need to be configured or an alternative mechanism such as an ETL will need to be employed. Linked schemas also can't be used to access data that is protected with PHI annotations.Lookup behavior: Lookups are removed from the source tables and queries when they are exposed in the linked schema. This prevents traversing a table in a linked schema beyond what has been explicitly allowed by the linked schema creator.URLs: URLs are also removed from the source tables. The insert, update, and delete URLs are removed because the linked schema is considered read-only. The details URL and URL properties on columns are removed because the URL would rarely work in the linked schema container. If desired, the lookups and URLs can be added back in the linked schema metadata xml. To carry over any attachment field links in the source table, first copy the metadata that enables the links in the source table and paste it into the analogous field in the linked schema table. See below for an example.
Create a Linked Schema
A user must have the "Site Administrator" role to create a linked schema.To create a linked schema in folder B that reveals data from folder A:
Navigate to folder B.
Select > Go To Module > Query.
Click Schema Administration.
Under Linked Schemas, click New Linked Schema and specify the schema properties:
Schema Name: Provide a name for the new schema.
Source Container: Select the source folder that holds the originating schema (folder A).
Schema Template: Select a named schema template in a module. (Optional.)
Source Schema: Select the name of the originating schema in folder A.
Published Tables: To link/publish all of the tables and queries, make no selection. To link/publish a subset of tables, use checkboxes in the multi-select dropdown.
Metadata: Provide metadata filters for additional refinement. (Optional.)
Metadata Filters
You can add metadata xml that filters the data or modifies how it is displayed on the page.In the following example, a filter is applied to the table Location such that a record is shown only when InUse is true.
Attachment fields in the source table are not automatically carried over into the target schema, but you can activate attachment fields by providing a metadata override. For example, the XML below activates an attachment field called "AttachedDoc" in the list called "SourceList", which is in the domain/folder called "/labkey/SourceFolder". The URL must contain the listId (shown here = 1), the entityId, and the name of the file.
The entityId is a hidden field that you can expose on the original list. To see it, open (Grid Views) > Customize Grid, then check the box for "Show Hidden Fields" and scroll to locate the checkbox for adding "Entity Id" to your grid. Click View Grid to see it. To include it in the URL carried to the target schema, use the syntax "${EntityId} as shown above.For more information about metadata xml, see Query Metadata.
Schema Template
Default values can be saved as a "schema template" -- by overriding parts of the template, you can change:
the source schema (for example, while keeping the tables and metadata the same).
the metadata (for example, to set up different filters for each client).
Set up a template by placing .template.xml file in the resources/schemas directory of a module:
myModuleA/resources/schemas/ClientA.template.xml
The example .template.xml file below provides a default linked schema and a default filter xml for Client A:ClientA.template.xml
To use the module, you must enable it in the source folder (folder A):
Go to the source folder and select > Folder > Management.
Select the Folder Type tab.
Under Modules, check the box next to your module.
Click Update Folder.
You can override any of the template values by clicking Override template value for the appropriate region.For example, you can create a schema for Client B by (1) creating a new linked schema based on the template for Client A and (2) overriding the metadata xml, as shown below:Once you've clicked override, the link will change to read Revert to template value giving you that option.
Troubleshooting Linked Schemas
Linked schemas will be evaluated with the permissions of the user who originally created them. If that user is later deleted from the system, or their permissions change, the schema may no longer work correctly. Symptoms may be errors in queries, the disappearance of the linked schema from the schema browser (while it still appears on the schema administration page) and other issues.There are two ways to fix this:
Recreate the linked schema with the same name as a validated user. An easy way to do this is to rename the 'broken' schema, then create a new one with the original name, using the 'broken' one as a guide.
Directly access the database and find the query.ExternalSchema table row for this specific linked schema. Replace the 'CreatedBy' user value from the deleted user to a validated user. Clear Caches and Refresh after making this change.
Data within LabKey is ordinarily contained and restricted for use in a specific folder. In cases when you want to provide wider data access across a site, this topic summarizes various options for making data more broadly available outside of a specific folder.
SQL queries have access to every container, allowing you to pull in data from anywhere on the server. For details see Queries Across Folders.Queries can also be made available in child folders: see Edit Query Properties.Custom Grids based on queries can also be made available in child folders: see Customize Grid Views.Folder Filters can expand the scope of a query to range over different containers: see Query Scope: Filter by Folder.
Linked Schemas
Linked schemas let you pull in data from an existing LabKey schema in a different folder. The linked schema may expose some or all of the tables and queries from the source schema. The linked tables and queries may be filtered such that only a subset of the rows and/or columns are available. Linked schemas are unique in that they can give users access to data they would not otherwise have permission to see. All the other approaches described here require the user to have permission to the container in which the data really lives. For details see Linked Schemas and Tables.Custom Grids based on linked schema queries can also be made available in child folders: see Customize Grid Views.
Lookups and Reports
Lookup columns let you pull in data from one table into another. For example, setting up a lookup to a list or dataset as the common source for shared content, can simplify data entry and improve consistency. Lookups can target any container on the site. See Lookup Columns.R Reports can be made available in child folders. For details see: Saved R Reports.
Studies
When you create a published study, LabKey Server creates a new folder (a child folder of the original study folder) and copies snapshots of selected data into it, assembling the data in the form of a new study.Publishing a study is primarily intended for presenting your results to a new audience. For details see Publish a Study.Sharing datasets and timepoints across multiple studies is an experimental feature. See Shared Datasets and Timepoints.
Assay Data
To give an assay design project-wide scope, create the design at the project level instead of specifying the subfolder. Project-level assay designs are available in every folder and subfolder of the project.For site-wide access, create the assay design in the Shared project.Assay data can also be made available in child folders: see Customize Grid Views.Assay runs and results can be linked to any study folder on the site (to which the linking user has permission). See Link Assay Data into a Study.
Sample Types
Create a Sample Type in the Shared project to have access to it site-wide, or create it in the project for access in that project's subfolders. See Samples.Sample data can be linked to any study folder on the site (to which the linking user has permission). See Link Sample Data to Study.
Collaboration Tools
Wiki pages defined in a given container can be reused and displayed in any other container on the server. For details see: Wiki Admin Guide.If you create an Issue Tracker definition in the Shared project, it will be available site wide. Created in the project, it will be available in all folders in the project.
Files
Folders store and access files on the file system via the file root. The default root is specific to the container, but if desired, multiple containers can be configured to use the same set of files from the file system using a custom pipeline root. See File Root Options.Using a Premium Edition of LabKey Server you can also configure one or more file watchers to monitor such a shared location for data files of interest.
Security
Scoping access to data through security groups and permissions is another way to control availability to your data. Set up site or project groups for your users, and choosing when to use inheritance of permissions in subfolders gives you many options for managing access.
You can also expose only a selected subset of your data broadly, while protecting PHI for example, using row and column level permissions as outlined here:
Premium Feature — Available in the Enterprise Edition of LabKey Server. Learn more or contact LabKey.
Ontologies help research scientists in many specialties reconcile data with common and controlled vocabularies. Aligning terms, hierarchies, and the meaning of specific data columns and values is important for consistent analysis, reporting, and cross-organization integration. An ontology system will standardize concepts, understand their hierarchies, and align synonyms which describe the same entities. You can think of an ontology like a language; when your data is all speaking the same language, greater meaning emerges.
The ontology module enables reporting and using controlled vocabularies. Many such vocabularies are in active use in different research communities. Some examples:
NCIT (NCI Thesaurus): A reference terminology and biomedical ontology
LOINC (Logical Observation Identifiers Names and Codes): Codes for each test, measurement, or observation that has a clinically different meaning
SNOMED_CT (Systemized Nomenclature of Medicine - Clinical Terms): A standardized multilingual vocabulary of clinical terminology used for electronic exchange of health information
MSH (Medical Subject Headings): Used for indexing, cataloging, and searching for health related information
The first step is to generate an ontology archive that can be loaded into LabKey. A set of python scripts is provided on GitHub to help you accomplish this.
The python script is designed to accept OWL, NCI, and GO files, but support is not universal for all files of these types. Contact your Account Manager to get started and for help with individual files.Once generated, the archive will contain individual text files:
concepts.txt: (Required) The preferred terms and their codes for this ontology. The standard vocabulary.
hierarchy.txt: (Recommended) The hierarchy among these standard terms, expressed in levels and paths to the codes. This can be used to group related items.
synonyms.txt: An expected set of local or reported terms that might be used in addition to the preferred term, mapping them to the code so that they are treated as the same concept.
Load and Use Ontologies
To learn about loading ontologies onto your LabKey Server and enabling their use in folders, see this topic: Load OntologiesOnce an ontology has been loaded and the module enabled in your folder, the field editor will include new options and you can use ontology information in your SQL queries.
Concept Annotations
Concept annotations let you link columns in your data to concepts in your ontology. The field editor will include an option to associate the column with a specific concept. For example, a "medication" column might map to a "NCIT:ST1000016" concept code used in a centralized invoicing system.Learn more in this topic: Concept Annotations
Ontology Lookup Fields
The field editor also includes an Ontology Lookup data type. This field type encompasses three related fields, helping you take an input value and look up both the preferred term and the code used in the ontology.Learn more in this topic: Ontology Lookup
Ontologies in SQL
New LabKey SQL syntax allows you to incorporate ontology hierarchy and synonyms into your queries.Learn more in this topic: Ontology SQL
Premium Feature — Available in the Enterprise Edition of LabKey Server. Learn more or contact LabKey.
This topic covers how to load ontology vocabularies onto your server and enable their use in individual folders. It assumes you have obtained or generated an ontology archive in the expected format. You also must have the ontology module deployed on your server.
One or more ontology vocabularies can be loaded onto your server. Ontologies are stored in the "Shared" folder where they are accessible site-wide. You may load ontologies from any location on the server.
Select > Go To Module > More Modules > Ontology.
You will see any ontologies already loaded.
Click Add LabKey Archive (.Zip) to add a new one.
Enter:
Name: (Required)
Abbreviation: (Required) This should be a short unique string used to identify this ontology. This value can't be changed later.
Description: (Optional)
Click Create.
On the next page, use Browse or Choose File to locate your archive.
Ontology zip archives include the files: concepts.txt, hierarchy.txt, synonyms.txt
Click Upload.
You will see the pipeline task status as the ontology is loaded. Depending on the size of the archive, this could take considerable time, and will continue in the background if you navigate away from this page.Once the upload is complete, return to > Go To Module > Ontology (you may need to click "More Modules" to find it).
Note that if you access the Ontology module from a folder other than /Shared, you will see "defined in folder /SHARED" instead of the manage links shown below. Click /Shared in that message to go to the manage UI described in the next section.
Manage Ontologies
In the Ontologies module in the /Shared project, you will see all available ontologies on the list.
Click Browse to see the concepts loaded for this ontology. See Concept Annotations.
Click Re-Import to upload an archive for the ontology in that row.
Click Delete to remove this ontology.
Click Browse Concepts to see the concepts loaded for any ontology.
Click Add LabKey Archive (.Zip) to add another.
Learn about browsing the concepts in your ontologies in this topic: Concept Annotations
Enable Ontologies in Folders
To use the controlled vocabularies in your data, enable the Ontology module in the folders where you want to be able to use them.
Navigate to the container where you want to use the ontology.
Select > Folder > Management and click the Folder Type tab.
Premium Feature — Available in the Enterprise Edition of LabKey Server. Learn more or contact LabKey.
Once ontologies have been loaded and enabled in your folder, you can use Concept Annotations to link fields in your data with their concepts in the ontology vocabulary. A "concept picker" interface makes it easy for users to find desired annotations.
Reach the grid of ontologies available by selecting > Go To Module > More Modules > Ontology.
Click Browse Concepts below the grid to see the concepts, codes, and synonyms loaded for any ontology.
On the next page, select the ontology to browse.
Note that you can shortcut this step by viewing ontologies in the "Shared" project, then clicking Browse for a specific row in the grid.
Type into the search bar to immediately locate terms. See details below.
Scroll to find terms of interest, click to expand them.
Details about the selected item on the left are shown to the right.
The Code is in a shaded box, including the ontology prefix.
Any Synonyms will be listed below.
Click Show Path or the Path Information tab to see the hierarchy of concepts that lead to the selection. See details below
Search Concepts
Instead of manually scrolling and expanding the ontology hierarchy, you can type into the search box to immediately locate and jump to concepts containing that term. The search is specific to the current ontology; you will not see results from other ontologies.As soon as you have typed a term of at least three characters, the search results will populate in a clickable dropdown. Only full word matches are included. You'll see both concepts and their codes. Click to see the details for any search result. Note that search results will disappear if you move the cursor (focus) outside the search box, but will return when you focus there again.Search terms will not autocomplete any suggestions as you type or detect any 'stem' words, i.e. searching for "foot" will not find "feet".
Path Information
When you click Show Path you will see the hierarchy that leads to your current selection.Click the Path Information for a more complete picture of the same concept, including any Alternate Paths that may exist to the selection.
Add Concept Annotation
Open the field editor where you want to use concept annotations. This might mean editing the design of a list or the definition of a dataset.
Expand the field of interest.
Under Name and Linking Options, click Select Concept.
In the popup, select the ontology to use. If only one is loaded, you will skip this step.
In the popup, browse the ontology to find the concept to use.
Click it in the left panel to see the code (and any synonyms) on the right.
Click Apply.
You'll see the concept annotation setting in the field details.
Save your changes.
View Concept Annotations
In the data grid, hovering over a column header will now show the Concept Annotation set for this field.
Edit Concept Annotation
To change the concept annotation for a field, reopen the field in the field editor, click Concept Annotation, make a different selection, and click Apply.
Premium Feature — Available in the Enterprise Edition of LabKey Server. Learn more or contact LabKey.
Ontology lookup columns support filtering based on concept and path within the ontology. Filtering based on whether a given concept is in an expected subtree (or not in an unexpected one) can isolate desired data using knowledge of the concept hierarchy.
To use the ontology tree browser, click the header for a column of type "Ontology Lookup" and select Filter.... You cannot use this filtering on the "import" or "label" fields related to your lookup, only the "code" value, shown here as "Medication Code".Select the Choose Filters tab, then select a concept using the Find <Column Name> By Tree link.The browser is similar to the concept browser. You can scroll or type into the "Search" bar to find the concept you want. Click the concept to see it within the hierarchy, with parent and first children expanded.When you locate the concept you want, hover to reveal a filter icon. Click it to place the concept, with code, in the filter box. when using the subtree filter expressions you'll see the path to the selected concept. Shown below, we'll filter for concepts in the subtree under "Analgesic Agent".Click Close Browser when finished. If needed, you can add another filter before saving by clicking OK.
Subtree Filter Expressions
The example above shows how a subtree filter value is displayed. Notice the slashes indicating the hierarchy path to the selected concept.
Is In Subtree: The filter will return values from the column that are in the subtree below the selected concept.
Is Not In Subtree: The filter will return values from the column that are in the subtree below the selected concept.
These subtree-based expressions can be combined to produce compound filters, such as medications that are "analgesic agents" but not "adjuvant analgesics":
Single Concept Filter Expressions
When using the Equals or Does Not Equal filtering expressions, browse the tree as above and click the filter icon. The code will be shown in the box.
Set of Concepts Filter Expressions
The filter expressions Equals One Of and Does Not Equal Any Of support multiselection of as many ontology concepts as necessary. Click the filter icons to build up a set in the box, the appropriate separator will be inserted.
Once an ontology has been loaded and the module enabled in your folder, the field editor will include an Ontology Lookup data type for all column types. Any column may be defined to use the standard vocabularies available in a selected ontology source.There can be up to three related fields in the data structure. The naming does not need to match these conventions, but it can be helpful to clarify which columns contain which elements:
columnName_code: The ontology standard code, a string which can either be provided or retrieved from the ontology. This column is configured as an Ontology Lookup and may have one or both of the following related columns:
columnName_import: (Optional) The reported, or locally used term that could be imported with the data. If no code is provided, the ontology lookup mechanism will seek a unique code value (and label) matching this term.
columnName_label: (Optional) The preferred or standard term retrieved from the ontology. This field is read-only and set by the ontology lookup mechanism as a user-readable supplement to the code value.
Lookup Rules
On data import (or update), the following ontology lookup actions will occur:
If only an "*_import" value is provided, the "*_label" and "*_code" will be populated automatically if found in the ontology as a concept or synonym.
Note that a code value can be provided as the "*_import" value and will be a successful lookup, provided it is included in the ontology as a synonym for itself.
If the provided "*_import" happens to match a concept "*_label", it will be shown in both columns: retained as the "*_import" and populated by the lookup as the "*_label".
If only a code value is provided directly in the "*_code" column, the label will be populated from the ontology. (No "*_import" value will be populated.)
Note that when the code value is provided in the "*_code" column, it must be prefixed with the ontology abbreviation. Ex: "NCI:c17113", not just "c17113".
If both an "*_import" and "*_code" value are provided (even if null), the "*_code" is controlling. Note that this means that if there is a "*_code" column in your data, it must contain a valid code value, otherwise the "null" value of this column will "overrule" the lookup of any "_import" value provided. This means:
For data insert or bulk update, you should include either the "*_code" column or the "*_import" column, but not both, unless the "*_code" is always populated.
For data update in the grid, providing an "*_import value will not look up the corresponding code; the "*_code" column is present (and null during the update), so will override the lookup.
Usage Example
For example, if you had a dataset containing a disease diagnosis, and wanted to map to the preferred label and code values from a loaded ontology, you could follow these steps:
Create three fields for your "columnName", in this case "Disease", with the field suffixes and types as follows:
Field Name
Data Type
Disease_import
Text
The term imported for this disease, which might be a synonym or could be the preferred term.
Disease_label
Text
The standard term retrieved from the ontology.
Disease_code
Ontology Lookup
Will display the standard code from the ontology and provides the interconnection
Expand the "columnName_code" field.
Under Ontology Lookup Options:
Choose an Ontology: The loaded ontologies are all available here. Select which to use for this field.
Choose an Import Field: Select the "columnName_import" field you defined, here "Disease_import"
Choose a Label Field: Select the "columnName_label" field you defined.
Save.
When data is imported, it can include either the "*_import" field or the "*_code" field, but not both. When using the "*_import" field, if
different terms for the same disease are imported, the preferred term and code fields will standardize them for you. Control which columns are visible using the grid customizer.Shown here, a variety of methods for entering a COVID-19 diagnosis were provided, including the preferred term for patient PT-105 and the code number itself for PT-104. All rows can be easily grouped and aggregated using the label and code columns. The reported "*_import" value is retained for reference.
Concept Picker for Insert/Update with Type-ahead
When an Ontology Lookup field, i.e. the "*_code" field, is included in the insert or update form, you will see a "Search [Ontology Name]" placeholder. Type ahead (at least three characters, and full words for narrower lists) to quickly browse for codes that match a desired term. You'll see both the terms and codes in a dropdown menu. Click to select the desired concept from the list.You'll see the code in the entry box and a tooltip with the preferred label on the right.Alternately, you can use the Find [column name] By Tree link to browse the full Ontology to find a desired code. Shown below, "Medication Code" is the Ontology Lookup, so the link to click reads Find Medication Code By Tree.Use the search bar or scroll to find the concept to insert. As you browse the ontology concepts you can see the paths and synonyms for a selected concept to help you make the correct selection. If the field has been initialized to an expected concept, the browser will auto scroll to it.Click Apply to make your selection. You'll see the code in the entry box and a tooltip with the preferred label on the right as in the typeahead case above.
Initialize Expected Vocabulary
For the Ontology Lookup field, you choose an Ontology to reference, plus optional Import Field, and Label Field. In addition you can initialize the lookup field with an Expected Vocabulary making it easier for users to enter the expected value(s).
Open the field editor, then expand the Ontology Lookup field.
Select the Ontology to use (if it is not already selected).
Click Expected Vocabulary to open the concept picker.
Search or scroll to find the concept of interest. Shown here "Facial Nerve Palsy" or "NCIT:C26769".
Click Apply.
The selected concept is now shown in the expanded field details.
Premium Feature — Available in the Enterprise Edition of LabKey Server. Learn more or contact LabKey.
Once ontologies are loaded and enabled, you can also make direct use of them in SQL queries. The syntax described in this topic helps you access the preferred vocabularies and wealth of meaning contained in your ontologies.
Returns true if conceptX is contained in a subtree rooted at the unique path that contains .../conceptA/conceptB/conceptParent/.
If there is no such unique path the method returns false.
If there is such a unique path, but conceptX is not in this tree the method returns false.
ConceptPath
Usage:
ConceptPath(conceptA,conceptB,...,conceptParent)
This method takes one or more arguments, and returns the unique hierarchy path that contains the provided concepts in sequence (no gaps). The return value comes from the ontology.hierarchy.path column.
This method will return null if no such path is found, or if more than one path is found.
Note that the hierarchy paths may not be user readable at all, and may be more likely to change than the concept codes which are very stable. So this usage is preferable to trying to use hierarchy paths directly.
Performance note: It is faster to ask if conceptX belongs to a subtree ending in conceptParent than it is to ask whether conceptX is a subclass of conceptParent.For performance we store all possible paths in the “subclass” hierarchy to create a pure tree, rather than a graph of subclass relations. This makes it much easier to answer questions like select all rows containing a ‘cancer finding’. This schema means that internally we are really querying the relationship between paths, not directly querying concepts. Therefore ConceptIsSubClass() is more complicated to answer than ConceptIsInSubtree().
@concept
The @concept annotation can be used to override the metadata of the column with a concept annotation.Usage:
SELECT 'Something' as "Finding" @concept=C3367 FROM T WHERE ...
table.findColumn
To find a column in a given table by concept, conceptURI, name, propertyuri, or obsolete name
, use findColumn on your table:
table.findColumn([columnProperties])
For example, if you've annotated a column with the concept coded "ONT:123", use the following to return the column with that concept:
SELECT MyTable.findColumn(@concept='ONT:123') FROM Project.MyStudy.study.MyTable
Examples
In these examples, we use a fictional ontology nicknamed "ONT". A value like "ONT:123" might be one of the codes in the ontology meaning "Pharma", for example. All SQL values are just string literals of the concept code, including a readable name in a comment in these examples is for clarity.Here, "ONT:123" (Pharma) appears in the hierarchy tree once as a root term; "ONT:382" (Ibuprofen, for example) appears twice in the hierarchy 'below' Pharma, and has a further child: "ONT:350" (Oral form ibuprofen). Other codes are omitted for readability:
The two expressions below are not semantically the same, but return the same result. The second version is preferred because it ensures that there is only one path being evaluated and might be faster.
The next two expressions do not return the same result. The first works as expected:
IsSubClassOf('ONT:350' /* Oral form ibuprofen */, 'ONT:382' /* Ibuprofen */)
IsInSubtree('ONT:350' /* Oral form ibuprofen */, ConceptPath('ONT:382' /* Ibuprofen */)
Since there is not a unique concept path containing 'ONT:382' (Ibuprofen), the value of ConceptPath('ONT:382' /* Ibuprofen */) is NULL in the second row. Instead, the following expression would work as expected, clarifying which "branch" of the path to use:
LabKey Server offers a wide range of options for performing quality control checks and cleanup on your data. Depending on the type of data structure you use, different options are available.
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Various metrics can be used to track quality and investigate trends in different types of data. In Luminex assays and Panorama folders, Levey-Jennings plots offer built in support. Quality Control (QC) Trend Reports are a way to perform this type of quality control on other types of data, granting the ability to track metrics over time to aid in spotting possible quality problems.QC Trend Reports can be defined on lists, study datasets, and some assays, including standard and file-based assays. You can also define trend reports via the schema browser granting you more autonomy over what is tracked and how. In addition to Levey-Jennings reports, you can plot Moving Range, Cumulative Sum (either variability or mean) and options are also available for normalizing the Y-axis to the percent of mean or standard deviations for some plot types.QC Trend Reports are available on servers with the Premium module installed.
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Various metrics can be used to track quality and investigate trends in different types of data. In Luminex assays and Panorama folders, Levey-Jennings plots offer built in support. Quality Control (QC) Trend Reports are a way to perform this type of quality control on other types of data, such as Standard and file-based assay data.
QC Trend Reports are enabled on servers with the Premium module installed.
Options for Defining QC Trend Reports
Open the UI for defining the QC trend report in one of three ways.
Option 1. Associate with a Standard or File-based Assay
If you will be running your QC Trend Reports over assay data, you can define them from the assay grid header menu. This option is supported for either the Standard Assay, or a file-based assay. Associate QC Trend Reports with assays as follows:
You can also define a QC Trend Report from the (Charts) menu of a data grid:
Open the (Charts and Reports) menu.
Select Create QC Trend Report.
Using this method will typically prepopulate the schema and query you are viewing when you open the QC Trend Report editor. Note that you must manually specify the Assay if you are using the report on assay data but creating it from the charts menu.Continue to define your QC trend report.
Option 3. Define in Schema Browser
If neither of the above options is suitable, you can also manually define the report in the schema browser as follows:
Select > Go To Module > Query.
Choose the "qctrendreport" schema.
Select the "QCTrendReport" query (it will be alphabetized with any existing QC trend reports).
Click View Data.
You'll see a listing of existing QC trend reports.
QC Trend reports are defined using a two-panel wizard.
Details
Provide the name and other details.
Enter a Label, which must be unique to the folder.
Select a Schema from the dropdown. This will populate the "Query" dropdown. If you started this report from a grid of data, the schema and query will be prepopulated with defaults.
Select the Query within the selected schema.
Select the View from any defined on that query.
If the Report should be associated with a specific Assay, select it here.
When attached to an assay, the report will be available via the "View QC Trend Report" menu in assay data headers.
If the assay design is defined in the "/Shared" project, you will also see this report available in other containers where the assay design is used.
Date: Choose among columns of type "Date" in the selected query and view. Used for sorting and joining rows in the report and with guide sets.
X-Axis Label: This column will be used for the tick mark display along the x-axis.
Graph Parameter(s): Select one or more columns to be used for grouping/filtering in the trend report. They will be shown in the UI in the order selected here.
When a user is generating a report, values selected for the first parameter will limit the values available for subsequent parameter(s) to only combinations that are valid.
Metric(s): Define the metrics to be tracked by the trend report. Options are the columns of type "Numeric", i.e. integers and floating point numbers.
For assays, there are both "Run" fields and "Data" fields. Metrics should be defined in the Run Fields section.
Click the section to open it.
Expand the desired field (shown here "Metric1").
Click Advanced Settings.
Check the box for Make this field available as a measure.
Click Apply and then Save.
Learn more about setting measures and dimensions here.
Edit QC Trend Reports
To edit a QC trend report, first open the grid for the report. For reports associated with assays, the report grid will be available on the View QC Trend Reports menu for that assay.
To edit, click the (Configure) icon above the report grid.
For other reports, reach the assay report page using the query schema browser:
Select > Go To Module > Query.
Choose the "qctrendreport" schema.
Select the "QCTrendReport" query.
Note that each QC Trend Report you have defined will be represented by a folder, each of which contains another "QCTrendReport" query. You want the top level "QCTrendReport" query.
Click View Data to see a list of the existing queries in the Label column.
Click the Label for your QC Trend report to open the report grid.
Use the (Configure) icon to edit.
Delete QC Trend Reports
To delete a QC trend report, click the (delete) icon above the report grid.
Share Reports with Other Users
When you define a QC Trend report, you will be able to see it on the (Charts and Reports) menu for the appropriate dataset, or on the View QC Trend Report menu above assay run data. To make the report available to others, check the Shared box.The following table shows the container permission roles required to perform various tasks on a shared vs. non-shared report.
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Various metrics can be used to track quality and investigate trends in different types of data. In Luminex assays and Panorama folders, Levey-Jennings plots offer built in support. Quality Control (QC) Trend Reports are a way to perform this type of quality control on other types of data, such as GPAT and file-based assay data. To define a QC report, use this topic: Define QC Trend Report.
When using a QC trend report associated with an assay design, you'll be able to open the assay results and select View QC Trend Report > View [the name of your report] to access it.The trend report shows the specified metrics in a grid.
Click View Graph to select among the graph parameters you specified in your report.
The value selected for the each parameter will limit the values available for subsequent parameter(s) to only combinations that are valid. For example, if the first parameter were "US State" and the second "City", only cities in the selected state would be listed.
Click View Report.
The content of the report depends on how it was defined. As an example, this report shows a Levey-Jennings plot.Choose other Graph Parameters and click View Report to alter the report shown.
Use QC Trend Reporting with Other Data
When the QC Trend report is not associated with an assay, you will access it from the (Charts and Reports) menu for the table or query from which you created it.You'll select among the graph parameters you specified in your report, with only valid combinations shown once you've started selecting parameters. For example, if the first parameter were "US State" and the second "City", only cities in the selected state would be listed. Click View Report as for an assay based report.
QC Plot Options
Within the trend report, select various Plot Options:
Metric: Select among metrics provided in your configuration.
Y-Axis Scale:
Linear
Log
Percent of Mean: (not supported in CUSUM plots)A guide set must be defined to use this Y-axis scale. Values are plotted as the percent of mean from the guide set.
Standard Deviations: (not supported in CUSUM plots)A guide set must be defined to use this Y-axis scale. Values are plotted as standard deviations according to values in the guide set.
Show Trend Line: Uncheck the box to remove the line if desired.
Plot Size: Use the pulldown to select a size:
Large - 1500x450
Default - 1000x300
QC Plot Type: Select among plot types provided using checkboxes.
QC Plot Types
Use checkboxes to select among plot types:
Levey-Jennings: Visualize and analyze trends and outliers.
Moving Range: Monitor process variation by using the sequential differences between values.
CUSUMm: Mean Cumulative Sum: Plots both positive and negative shifts in the mean cumulative sum of deviation for monitoring change detection.
CUSUMv: Variability or Scale Cumulative Sum: Plots both positive and negative shifts in the variability of the given metric.
Multiple plots are stacked and labelled for clarity, as shown above. The legend clarifies which line is positive (dashed) and negative (solid) in each CUSUM plot.
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Various metrics can be used to track quality and investigate trends in different types of data. Using guide sets with QC Trend Reports helps you build expected value ranges into your QC reports.
For each combination of graph parameters, you can define guide sets for tracking performance over time. To define a guide set, click Create New. Guide sets are defined based on specified values per metric (i.e. mean and standard deviation). You specify the start date. If there is an existing guide set with a start date earlier than the one you provide for the new set, the end date of the old set will be set accordingly.Guide sets are displayed in the QC Trend Report graph as bar lines: the center is the mean, and 3 standard deviations above and below are marked. Any outliers are shown with a colored highlight and listed under Tracking Data as outliers for the specific metric/run combination.There can be more than one guide set defined and in use simultaneously. Shown above, there are two: a previous set applied to Run01 and Run02, and the most recent guide set which is applied to runs 3 and 4. Outliers are shown in red and at the far right of the Tracking Data column, there are columns indicating these outliers for each metric. Notice that whether a point is an outlier or not is determined based on the guide set applied to that run, not necessarily the most recent guide set. In the above illustration, the point for run01.xls would not be an outlier under the most recent guide set, but is under the earlier guide set which is applied to run1.
Edit Guide Sets
You can edit the most recently defined guide set, which is identified under the plot area, directly from the user interface by clicking Edit.The interface is the same as for new guide set creation: Change or enter the desired values in the Mean and Std Dev boxes, then click Submit to save your changes.To edit any other guide sets, click Edit/Delete Related Guide Sets. This page will open in a new browser tab (allowing you to return to the report itself without reentering the parameters.Hover over the row of interest to expose the icons, then click the (pencil) icon to edit that guide set. The format for editing older guide sets is less graphical, but by modifying the JSON in the Configuration field, you can still make the same changes to the Mean and Std Dev settings for the metrics.Click Submit to save your changes. Reopen to the report to view the newly changed guide set.
Delete Guide Sets
To delete a guide set, click Edit/Delete Related Guide Sets to open the list. Select the row to delete and click the (delete) icon. Deleting a guide set cannot be undone.
View Multiple Guide Sets
When more than one guide set is defined, you can review them by clicking the link View Related Guide Sets under the plot.The guide sets are listed with start dates, and end dates if applicable, as well as the mean and standard deviation settings for each metric.
LabKey provides full-text search across data in your server. Search is secure, so you only see results that you have sufficient permissions to view. Results are ordered by relevance.
The site wide search on a LabKey Server is available by clicking the icon in the header bar. A project or folder wide search box can be provided in a web part.
If you are looking for details about search in LabKey Sample Manager and LIMS products, see these topics:
Not all features described in this topic are available in those products.
Search Terms and Operators
To search, enter terms (search words) and operators (search modifiers) in the search box using the following guidelines:Terms
The words or phrases you enter are searched as if they have "OR" between them by default. This means results returned will include at least one of the terms.
Example: a search for NAb assay returns all pages that contain at least one of the terms "NAb" and "assay". Pages that contain both will appear higher in the results than pages that contain just one of the terms.
Double quotes around phrases indicate that they must be searched as exact phrases instead of as individual terms.
Example: Searching the quoted phrase "NAb assay" returns only pages that include this two word phrase.
Operators
AND: Use the AND operator to limit results to pages with both terms.
Example: NAb AND assay returns all pages that contain the term both the term "NAb" and the term "assay", in any order.
+: A search term preceded by the + operator must appear on returned pages.
Example: NAb +assay returns pages that must contain the term "assay" and may contain the term "NAb".
NOT: Use NOT before a search term that must not appear on returned pages.
Example: NAb NOT assay returns pages that contain the term "NAb" but do not contain the term "assay".
-: Use like the NOT operator; a term preceded by - must not appear on returned pages.
Example: NAb -assay returns all pages that contain the term "NAb" but do not contain the term "assay".
Note: If you use a '-' hyphen as a separator within names of elements, such as when using a sample naming pattern, be aware that you need to surround a search for one of these elements with double quotes. The hyphen will be interpreted as a minus, or NOT operator, and the search string will be broken into two parts. For example:
Searching for "Sample-11" will find the sample named Sample-11.
Searching for Sample-11 without the quotes will return results that contain "Sample" and not "11", i.e. may not find the desired sample, depending on whether it also contains "11" as a string, and will return unintended results that do not match "Sample-11".
Other Guidelines
Capitalization is ignored.
Parentheses can be used to group terms.
Extraction of root words, also known as stemming, is performed at indexing and query time. As a result, searching for "study", "studies", "studying", or "studied" will yield identical results.
Wild card searches
Use the question mark (?) for single character wild card searches. For example, searching for "s?ed" will return both "seed" and "shed".
Use the asterisk character (*) for multiple character wild card searches. For example, searching for "s*ed" will return both "seed" and "speed".
Wild card searches cannot be used as the start of the search string. For example, "TestSearch*" is supported, but "*TestSearch" is not.
Note that stemming (defined above) creates indexes only for root words, so wild card searches must include the root word to yield the intended results.
Content Searched
Data types and sources. The LabKey indexer inventories most data types on your server:
Study protocol documents and study descriptions.
Study dataset metadata (study labels; dataset names, labels, and descriptions; columns names, labels, and descriptions; lab/site labels)
List metadata and/or data, including attached documents. You have precise control of which parts of a list are indexed. For details see Edit a List Design.
Schema metadata, including external schema
Data Classes, including Registry and Media entities in Biologics and Source Types in Sample Manager (name, description of both the class and class members are searchable)
Participant IDs
Wiki pages and attachments
Messages and attachments
Issues
Files
Automatically includes the contents of all file directories. File directories are where uploaded files are stored by default for each folder on your server. See also: File Terminology
By default, does not include the contents of pipeline override folders (@pipeline folders). The contents of these folders are only indexed when you set the indexing checkbox for the pipeline override that created them. Files are only stored in pipeline override folders when you have set a pipeline override and thus set up a non-default storage location for uploaded files.
Folder names, path elements, and descriptions. A separate "Folders" list is provided in search results. This list includes only folders that have your search term in their metadata, not folders where your search term appears in content such as wiki pages.
File formats. The indexer can read attachments and files in a variety of document formats, including: HTML, XML, text, Microsoft Office (both the legacy binary and newer XML formats used by Excel, Word, PowerPoint, and Visio), OpenDocument (used by OpenOffice), RTF, PDF, and FCS (flow cytometry data files). Metadata at the top of MAGE-ML, mzXML and mzML files are searched, but not the full text of these types of files. The indexer does not read the contents of .zip archives.
Scoping
The search box in the header bar searches across the entire site. Search boxes within particular folders search only within that particular container by default. They can optionally be set by an admin to search subfolders within that container.Using advanced search options allows users to broaden or narrow a search.Search results are always limited to content the user has permission to view.
Advanced Search Options
Change the scope to search and type of results to return using advanced search options. First perform a search, then click advanced options.Select a Scope to scope your search to the contents of the entire site, the contents of the current project, the contents of the current folder without it sub-folders, or the contents of the current folder including its sub-folders.Choose one or more Categories to narrow your search to only certain result types. For example, if you select Files and Wikis you will no longer see datasets or messages in your results.Choose a Sort for search results. Options are: relevance, created, modified, and folder path. Check the box to reverse the direction of the sort in your results.
Search URL Parameters
You can define search parameters directly in the URL, for example, the following searches for the term "HIV" in the current folder:
You can assign multiple values to a parameter using the plus sign (+). For example, the following searches both files and wikis for the search terms 'HIV' and 'CD4':
?q=HIV+DC4&category=File+Wiki
Exact search phrases are indicated with quotes. The following URL searches for the phrase "HIV count":
Project, Folder, FolderAndSubfolders. No value specified searches the entire site.
showAdvanced
When the search results are returned, determines whether the advanced options pane is displayed.
true, false
Example: Participant Search
It's easy to see all of the data available for a particular individual across all studies on a LabKey Server -- just enter the participant ID into the search bar. The appropriate participant page, attachments, and other documents that mention this participant will be shown. You will only see materials you are authorized to view.
Related Topics
Lucene Search Syntax : Additional query syntax that can be used for wildcard searches, fuzzy searches, proximity searches, term boosting, etc.
An administrator can configure the location of the full-text search index and related settings. Start and pause the crawler and even delete and reindex the entire system if necessary. The Audit Log captures both the searches performed by users and any activities like reindexing.
The Full-Text Search Configuration page allows you to configure the index and review statistics about your index.
Go to > Site > Admin Console.
In the Management section, click Full-text search.
Index Configuration
Setting the Path. You can change the directory that stores the index by entering a new path and clicking Set Path. The default location is:
<LABKEY_HOME>/files/@labkey_full_text_index
The path can include substitution tokens, such as the server GUID or version.
Hover over the ? to see a full list of available tokens and usage examples.
You'll see what the path would resolve to (i.e. current values of any substitution tokens will be shown).
Specifying a non-default index path and/or using substitution tokens is especially useful if you are running multiple LabKey deployments on the same machine, because it allows each LabKey deployment to use a unique index.
Changing the location of the index requires re-indexing all data, which may affect performance.
When the index is created, the server GUID is written into it so that on later restarts, the server can detect whether the index at the provided path was created for itself or for another server. If the GUID does not match, the server will delete and reindex.
Developers who want to avoid this reindexing on a local machine can do so by using substitution tokens in the index path.
Be sure to place your @labkey_full_text_index on a drive with sufficient space.
We recommend never placing the search index on an NFS drive or AWS EFS. Learn more hereand here.
Other actions and settings on this page include:
Start/Pause Crawler. The crawler, or document indexer, continually inventories your site when running. You might pause it to diagnose issues with memory or performance.
Delete Index. You can delete the entire index for your server. Please do this with caution because rebuilding the index can be slow.
Directory Type. This setting lets you can change the search indexing directory type. The setting marked "Default (MMapDirectory)" allows the underlying search library to choose the directory implementation (based on operating system and 32-bit vs. 64-bit). The other options override the default heuristic and hard-code a specific directory implementation. These are provided in case the "Default" setting causes problems on a specific deployment. Use the default type unless you see a problem with search. Contact LabKey for assistance if full-text indexing or searching seems to have difficulty with the default setting.
Indexed File Size Limit: The default setting is 100 MB. Files larger than the limit set on this page will not be indexed. You can change this cap, but this is generally not recommended. Increasing it will result in additional memory usage; if you increase beyond 100MB, we recommend you also increase your heap size to be 4GB or larger. The size of xlsx files is limited to 1/5 the total file size limit set (i.e. defaults to 20MB).
This section provides information on the documents that have been indexed by the system, plus identifies the limits that have been set for the indexer by the LabKey team. These limits are used to manage index size and search precision. For example, you will see the "Maximum Size" of files that will be scanned by the indexer; the maximum size allows the system to avoid indexing exceptionally large files. Indexing large files will increase the index size without adding much (if any) value to searches.
Search Statistics
Lists the average time in milliseconds for each phase of searching the index, from creating the query to processing hits.
Audit Log
To see the search audit log:
Go to > Site > Admin Console.
In the Management section, click Audit Log.
Choose the Search option in the pulldown menu.
This displays the log of audited search events for your system. For example, you can see the terms entered by users in the search box. If someone has deleted your search index, this event will be displayed in the list, along with information on the user who ordered the delete.
Set Up Folder-Specific Search Boxes
By default, a site-wide search box is included in the LabKey header. You can add additional search boxes to individual projects or folders and choose how they are scoped.
This search will only search the container where you created it.
To also include subfolders, select Customize from the (triangle) menu and check the box to "Search subfolders".
As an example of a search box applied to a particular container, use the search box to the right of this page you are reading. It will search only the current folder (the LabKey documentation).
List and External Schema Metadata
By default, the search index includes metadata for lists and external schemas (including table names, table descriptions, column names, column labels, and column descriptions).You can control indexing of List metadata when creating or editing a list definition under Advanced List Settings > Search Indexing Options. Learn more here: Edit a List DesignYou can turn off indexing of external schema metadata by unchecking the checkbox Index Schema Meta Data when creating or editing an external schema definition. Learn more here: External Schemas and Data Sources
Include/Exclude a Folder from Search
You may want to exclude the contents of certain folders from searches. For example, you may not want archived folders or work in progress to appear in search results.
To exclude the contents of a folder from searches:
Navigate to the folder and select > Folder > Management.
Click the Search tab.
Uncheck the checkbox Include this folder's contents in multi-folder search results.
Click Save.
Note that this does not exclude the contents from indexing, so searches that originate from that folder will still include its contents in the results. Searches from any other folder will not, even if they specify a site- or subfolder-inclusive scope.
Exclude a File/Directory from Search Indexing
LabKey generally indexes the file system directories associated with projects and folders, i.e. the contents of the @files and other filesets. Some file and directory patterns are ignored (skipped during indexing), including:
Contents of directories named ".Trash" or ".svn"
Files with names that start with "."
Anything on a path that includes "no_crawl"
Contents of any directory containing a file named ".nocrawl"
On postgres contents of a directory containing a file named "PG_VERSION"
To exclude a file or the content of a folder from indexing, you may be able to employ one of the above conventions.
Export Index Contents
To export the contents of the search index:
Go to > Site > Admin Console.
Under Management click Full-Text Search.
At the bottom of the panel Index Statistics click Export Index Contents.
You can export the index in either Excel or TXT file formats.
Troubleshoot Search Indexing
Search Index not Initialized
When the search index cannot be found or properly initialized, you may see an error similar to the following. The <ERROR> can vary here, for example "Module Upgrade : Error: Unable to initialize search index." or "SearchService:index : Error: Unable to initialize search index."
ERROR LuceneSearchServiceImpl <DATE> <ERROR> Unable to initialize search index. Search will be disabled and new documents will not be indexed for searching until this is corrected and the server is restarted.
Options for resolving this include:
If you know the path is incorrect, and there is a path to where the .tip file resides provided, you can correct the Path to full-text search index and try again.
You can also Delete the current index and start the crawler again, essentially re-indexing all of your data. This option may take time but will run in the background, so can complete while you do other work on the server.
Search Index Corrupted (write.lock file)
If the labkey-errors.log includes an error like:
org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: <FULL_PATH_TO>files@labkey_full_text_indexwrite.lock
This indicates the index was corrupted, possibly because of an interrupted upgrade/reindex or problem connecting to the underlying file system during a previous reindexing.To resolve, first "Pause Crawler", then "Delete Index". Now restart the system and wait for it to come on line fully for users. Once it is back online, return to the admin dashboard to "Start Crawler", noting that this may take significant time to rebuild the index.
Threads Hang During Search
If you experience threads hanging during search operation, check to ensure that your search index is NOT on an NFS filesystem or on AWS EFS. These filesystems should never be used for a full-text search index.
No Search Results When Expected
If you encounter unexpected search results, particularly a lack of any results when matches are known, you may need to rebuild the search index. This situation may occur in particular on a development machine where you routinely pause the crawler.To rebuild the index, pause the crawler (if running), delete the index, and restart the crawler to and rebuild it.
Premium Feature — Available in the Professional and Enterprise Editions of LabKey Server. Also available as an Add-on to the Starter Edition. Learn more or contact LabKey.
LabKey's JDBC driver allows Spotfire to query the server and use the results to build visualizations and perform other Spotfire analysis.
Copy the LabKey JDBC driver jar to the appropriate location on the Spotfire Server.
Starting with Spotfire Server 10.3, the driver should be placed in the following directory, to ensure that the files are handled properly by the Spotfire Server Upgrade Tool:
<tibco-server-install>/tomcat/custom-ext/
On earlier versions of Spotfire, you may have placed the driver in the directory:
<tibco-server-install>/tomcat/lib/
If files are placed in this "lib" (or another) directory, then you will have to manually move them during an upgrade.
For any version, if you encounter problems such as "The information link cannot be loaded from the server", trying the other folder location may resolve the problem.
The connection URL pattern includes your HOST and PORT, as well as a CONTEXT_PATH if needed (uncommon). The folder path on LabKey Server will be appended to this connection URL pattern.For example, the connection URL line might look like:
To control what folders and subfolders of data will be queried, limiting (or expanding) the scope of data included in Spotfire visualizations, apply a container filter. Include a <connection-property> key-value pair in your XML configuration similar to this:
Possible values for the containerFilter parameter are:
Current (Default)
CurrentAndSubfolders
CurrentPlusProject
CurrentAndParents
CurrentPlusProjectAndShared
AllFolders
Learn more about using folder filters in this topic: filterbyfolder.
Using LabKey Data
In the Spotfire Information Designer add a data source of type 'labkey'. (Once you have successfully added the driver above, 'labkey' will be available in the Type dropdown.
You must also provide a Connection URL, Username, and Password. See an example screen shot below.
Now you will be able to see the LabKey Server folder hierarchy and exposed schemas.
Logging
In your Spotfire Server, you can add additional logging at these levels:
org.labkey.jdbc.level = FINE (Provides java.util.logging levels)
org.labkey.jdbc.level = DEBUG (Provides log4j or slf4j logging levels)
These will result in logging into the stderr log file appender.
Spotfire versions 7.9 and above use the log4j2 logging framework. This property should be set in the [Spotfire_Server_installation_dir]/tomcat/spotfire-config/log4j2.xml file. Generally there will already be a <Loggers> section in this file to which you can add:
In older versions of Spotfire prior to 7.9, loggers would have been included in a .properties file . Instructions for migrating a customized .properties file to the new format are available here:
If you have difficulties connecting to LabKey from Spotfire, first check the installation instructions in this topic and in LabKey JDBC Driver to ensure the properties and settings are configured correctly.Additional troubleshooting steps:
Confirm that the user account connecting from Spotfire can independently access the desired resources on LabKey Server directly.
Try accessing the same resources from Spotfire using a user account with site administrator access. If this works and the intended user does not, you may need to temporarily increase the account's permissions, or obtain an updated JDBC driver.
Premium Feature — Available in the Professional and Enterprise Editions of LabKey Server. Also available as an Add-on to the Starter Edition. Learn more or contact LabKey.
AWS Glue is a serverless ETL engine that can integrate data across many sources, including LabKey Server. LabKey Server implements the Postgres network wire protocol and emulates the database's responses, allowing supported tools to connect via standard Postgres ODBC and JDBC drivers. Clients connect to LabKey Server, not the backend database, ensuring consistent handling of data and standard LabKey user permissions. This means that you can use the Postgres JDBC driver that is preconfigured within Glue to connect to LabKey as a data source.
The connectors module must be deployed on your LabKey Server, and External Tool Connections must be enabled and accepting connections.
Overview of AWS Glue Setup
Your AWS Glue jobs need to run from a private subnet that is allowed to connect to your data sources (Intranet or Internet) via a NAT Gateway.
Create a VPC and set a tag name to help identify the resources being provisioned.
Create a NAT Gateway to let your Glue jobs connect to the outside world.
Identify a security group that allows all incoming and outgoing connections. It may be automatically created for you.
Create an IAM role with AWS Service as the Trusted Entity Type and select Glue from the "Use Cases for Other AWS Services" drop down. It should have the permission policies "AWSGlueServiceRole" and "AmazonS3FullAccess".
Create JDBC Connection
From the main AWS Glue landing page, set up the Connection (under Data Catalog).
Name: Enter a unique name.
Connection Type: Choose JDBC.
JDBC URL: Use the appropriate hostname, port, and container path based on your server's External Tool Connections configuration. Users can check the proper settings via the (username) > External Tool Access menu. It will start with:
Choose the VPC you are using, its private subnet, and a security group that allows incoming and outgoing traffic. In this UI, you'll only see the uniqueID for the resources, so cross-reference with the VPC section of the AWS Console.
Create a Target "Database"
The target database is where data lands when doing an ETL or a schema crawl.
From the AWS Glue main console, click the Databases link under the Data Catalog menu.
Click Add Database.
The Name you provide must be all lower case.
Click Create Database.
AWS Glue Crawler
A crawler connects to a data source and examines its available schemas and tables, making them available to create ETL jobs.
From the AWS Glue landing page, click Crawlers under Data Catalog.
Click Create Crawler.
Give it a Name and click Next.
Select the "Not yet" option for whether the data is already mapped to Glue tables.
Click Add a Data Source.
Data Source: Choose "JDBC" and select the Connection you created earlier.
Include Path: Point it at the desired container (exposed as a database)/schema/table. For example, to tell it to crawl all tables in the /Home project's core schema, use "home/core/%". Because the crawler uses slashes to separate the database/schema/table names, use a backslash to separate LabKey folders if you are not targeting a top-level project. For example, "home subfolder/core/%" will reference the /home/subfolder folder.
Click Add a JDBC Data Source.
Click Next.
Choose the IAM role you created earlier. The list of roles should be filtered to just those that have Glue permissions.
Click Next.
Choose the target database that you created earlier. Leave the Frequency set to "On demand".
Click Next.
Review your selections, then click Create Crawler.
Run the Crawler
If you just created the crawler, you'll be on its details page. If not, go to the AWS Glue console main menu, click the Crawlers link under the Data Catalog section, and click on your crawler.Click Run Crawler in the upper right.You can confirm that it visits the tables in the target schema by checking the Cloudwatch logs.
Create an ETL Job
First, create an S3 Bucket to hold the files that Glue creates. Make sure it is created in the same AWS region as your VPC.
Go to the main S3 Console and click Create Bucket.
Give it a name and either accept other default values or provide values as appropriate.
Click Create Bucket.
Next, create a Glue Job:
Go back to the main Glue page.
Click Jobs under the Data Integration and ETL heading.
Keep it simple by using the "Visual with a source and target" option.
Choose "PostgreSQL" as the source type and Amazon S3 as the target.
Click Create.
You should see a simple flowchart representing your job.
Click on the first node, labeled as "PostgreSQL table".
Under the Node Properties tab on the right, choose "Data Catalog table" as the JDBC source.
Select your "database" from the dropdown and then a table within it.
Click on the middle node, ApplyMapping.
Under the Transform tab on the right, it should show all of the columns in the table.
Click on the final step in the flowchart, S3 Bucket.
Select the Data Target Properties - S3 tab on the right.
Choose your preferred output format.
Click Browse S3. You should see the bucket you created. Select it, but don't click into it, because it will be empty unless you created a directory structure separately.
Now you can name and run your new job:
In the header, rename the job from "Untitled job" to your desired name.
Under the Job details table, choose the AIM Role you created.
Click Save, then click Run.
You can now navigate to the job details panel. After a few minutes, the job should complete successfully. When you go to your S3 bucket, the files created will be present.
Premium Feature — Available in the Enterprise Edition of LabKey Server. Learn more or contact LabKey.
Large amounts of clinical data is locked up in free-hand notes and document formats that were not originally designed for entry into computer systems. How can this data be extracted for the purposes of standardization, consolidation, and clinical research? LabKey's Natural Language Processing (NLP) and Document Abstraction, Curation, and Annotation workflow tools help to unlock this data and transform it into a tabular format that can better yield clinical insight.LabKey Server's solution focuses on the overall workflow required to efficiently transform large of amounts of data into formats usable by researchers. Teams can take an integrated, scalable approach to both manual data abstraction and automated natural language processing (NLP) engine use.LabKey NLP is available for subscription purchase from LabKey. For further information, please contact LabKey.
Premium Feature — Available in the Enterprise Edition of LabKey Server. Learn more or contact LabKey.
Natural Language Processing (NLP) techniques are useful for bringing free text source information into tabular form for analysis. The LabKey NLP workflow supports abstraction and annotation from files produced by an NLP engine or other process. This topic outlines how to upload documents directly to a workspace.
Another upload option is to configure and use a Natural Language Processing (NLP) pipeline with an integrated NLP engine. Different NLP engines and preprocessors have different requirements and output in different formats. This topic can help you correctly configure one to suit your appplication.
Accept other defaults in the folder creation wizard.
The default NLP folder contains web parts for the Data Pipeline, NLP Job Runs, and NLP Reports. To return to this main page at any time, click the Folder Name link near the top of the page.
Upload Documents Directly
Documents for abstraction, annotation, and curation can be directly uploaded. Typically the following formats are provided:
A TXT report file and a JSON results file. Each named for the document, one with with a .txt extension and one with a .nlp.json extension.
This pair of files is uploaded to LabKey as a directory or zip file.
A single directory or zip file can contain multiple pairs of files representing multiple documents, each with a unique report name and corresponding json file.
Configure a Natural Language Processing (NLP) Pipeline
An alternate method of integration with an NLP engine can encompass additional functionality during the upload process, provided in a pipeline configuration.
Setup the Data Pipeline
Return to the main page of the folder.
In the Data Pipeline web part, click Setup.
Select Set a pipeline override.
Enter the primary directory where the files you want to process are located.
Set searchability and permissions appropriately.
Click Save.
Click NLP Dashboard.
Configure Options
Set Module Properties
Multiple abstraction pipelines may be available on a given server. Using module properties, the administrator can select a specific abstraction pipeline and specific set of metadata to use in each container.These module properties can all vary per container, so for each, you will see a list of folders, starting with "Site Default" and ending with your current project or folder. All parent containers will be listed, and if values are configured, they will be displayed. Any in which you do not have permission to edit these properties will be grayed out.
Navigate to the container (project or folder) you are setting up.
Select > Folder > Management and click the Module Properties tab.
Under Property: Pipeline, next to your folder name, select the appropriate value for the abstraction pipeline to use in that container.
Upload the "metadata.json" file to the Files web part.
Select > Folder > Management and click the Module Properties tab.
Under Property: Metadata Location, enter the file name in one of these ways:
"metadata.json": The file is located in the root of the Files web part in the current container.
"subfolder-path/metadata.json": The file is in a subfolder of the Files web part in the current container (relative path)
"full path to metadata.json": Use the full path if the file has not been uploaded to the current project.
If the metadata file is in the files web part of the current container (or a subfolder within it), check the box for the folder name under Property: Is Metadata Location Relative to Container.
Define Pipeline Protocol(s)
When you import a TSV file, you will select a Protocol which may include one or more overrides of default parameters to the NLP engine.
If there are multiple NLP engines available, you can include the NLP version to use as a parameter. With version-specific protocols defined, you then simply select the desired protocol during file import. You may define a new protocol on the fly during any tsv file import, or you may find it simpler to predefine one or more. To quickly do so, you can import a small stub file, such as the one attached to this page.
Download this file: stub.nlp.tsv and place in the location of your choice.
In the Data Pipeline web part, click Process and Import Data.
Drag and drop the stub.nlp.tsv file into the upload window.
For each protocol you want to define:
In the Data Pipeline web part, click Process and Import Data.
Select the stub.nlp.tsv file and click Import Data.
Select "NLP engine invocation and results" and click Import.
From the Analysis Protocol dropdown, select "<New Protocol>". If there are no other protocols defined, this will be the only option.
Enter a name and description for this protocol. A name is required if you plan to save the protocol for future use. Using the version number in the name can help you easily differentiate protocols later.
Add a new line to the Parameters section for any parameters required, such as giving a location for an alternate metadata file or specifying the subdirectory that contains the intended NLP engine version. For example, if the subdirectories are named "engineVersion1" and "engineVersion2", you would uncomment the example line shown and use:
Select an Abstraction Identifier from the dropdown if you want every document processed with this protocol to be assigned the same identifier. "[NONE]" is the default.
Confirm "Save protocol for future use" is checked.
Scroll to the next section before proceeding.
Define Document Processing Configuration(s)
Document processing configurations control how assignment of abstraction and review tasks is done automatically. One or more configurations can be defined enabling you to easily select among different settings or criteria using the name of the configuration. In the lower portion of the import page, you will find the Document processing configuration section:
From the Name dropdown, select the name of the processing configuration you want to use.
The definition will be shown in the UI so you can confirm this is correct.
If you need to make changes, or want to define a new configuration, select "<New Configuration>".
Enter a unique name for the new configuration.
Select the type of document and disease groups to apply this configuration to. Note that if other types of report are uploaded for assignment at the same time, they will not be assigned.
Enter the percentages for assignment to review and abstraction.
Select the group(s) of users to which to make assignments. Eligible project groups are shown with checkboxes.
Confirm "Save configuration for future use" is checked to make this configuration available for selection by name during future imports.
Complete the import by clicking Analyze.
In the Data Pipeline web part, click Process and Import Data.
Upload the "stub.nlp.tsv" file again and repeat the import. This time you will see the new protocol and configuration you defined available for selection from their respective dropdowns.
Repeat these two steps to define all the protocols and document processing configurations you need.
While the engine is running, the pipeline web part will show a job in progress. When it completes, the pipeline job will disappear from the web part.
Refresh your browser window to show the new results in the NLP Job Runs web part.
View and Download Results
Once the NLP pipeline import is successful, the input and intermediate output files are both deleted from the filesystem.The NLP Job Runs lists the completed run, along with information like the document type and any identifier assigned for easy filtering or reporting. Hover over any run to reveal a (Details) link in the leftmost column. Click it to see both the input file and how the run was interpreted into tabular data.During import, new line characters (CRLF, LFCR, CR, and LF)are all normalized to LF to simplify highlighting text when abstracting information.
Note: The results may be reviewed for accuracy. In particular, the disease group determination is used to guide other values abstracted. If a reviewer notices an incorrect designation, they can edit, manually update it and send the document for reprocessing through the NLP information with the correct designation.
Download Results
To download the results, select (Export/Sign Data) above the grid and choose the desired format.
Rerun
To rerun the same file with a different version of the engine, simply repeat the original import process, but this time choose a different protocol (or define a new one) to point to a different engine version.
Error Reporting
During processing of files through the NLP pipeline, some errors which occur require human reconcilation before processing can proceed. The pipeline log is available with a report of any errors that were detected during processing, including:
Mismatches between field metadata and the field list. To ignore these mismatches during upload, set "validateResultFields" to false and rerun.
Errors or excessive delays while the transform phase is checking to see if work is available. These errors can indicate problems in the job queue that should be addressed.
Add a Data Transform Jobs web part to see the latest error in the Transform Run Log column.For more information about data transform error handling and logging, see ETL: Logs and Error Handling.
Premium Feature — Available in the Enterprise Edition of LabKey Server. Learn more or contact LabKey.
A metadata file written in JSON is used to configure the fields and categories for document abstraction. The file configures the field categories and lists available, as well as what type of values are permitted for each field.Metadata JSON files may be used to control a variety of implementation specific configurations, such as understanding common fields of a specific type of cancer report or case file. Implementing conditional or branching logic (if this is true, show these extra fields) can also be included using metadata.json files.
The metadata JSON file defines fields for abstraction using these properties:
field: (Required/String) The field name.
datatype: (Required/String) The type of the field.
closedClass: (Optional/Boolean) If true, the field is closed (only values on a pulldown menu can be selected). Otherwise and by default, arbitrary values can be entered into the field, whether or not they are on the expected list.
diseaseProperties: (Optional/Array of properties) The group of disease diagnoses for which this field is presented to abstractors. For example, if the field is shown for all types of disease, with four options on a dropdown list, the field might look like:
multiValue: (Optional/Boolean) If true, the field supports entry of multiple values. Otherwise, and by default, each field accepts a single value.
hidden: (Optional/Boolean) If true, the field is hidden in the initial state and only shown if triggered by conditional logic. Otherwise and by default, the field will be shown to the abstractor from the beginning.
disabled: (Optional/Boolean) If true, the field is shown, but disabled (grayed-out) in the initial state and only enabled to receive input via use of conditional logic. Otherwise and by default, the field will be enabled from the beginning.
Here, a Pathology category is defined, with table and field level groupings, and two tables with simple fields, "ClassifiedDiseaseGroup" and "Field1", a boolean.
To allow an abstractor to multiselect values from a pulldown menu, which will be shown in the field value separated by || (double pipes), include the following in the field definition:
"multiValue" : "True"
Conditional or Branching Logic
Conditional or branching logic can be used to control the fields and options presented to annotators. For example, options could vary widely by disease group or whether metastasis is reported. To introduce conditional logic, use listeners and handlers within the metadata JSON file.To enable or disable the field upon initial opening of the abstraction/annotation screen, use the disabled boolean described above.
Listener Configurations
A listener configuration includes the following properties:
field: (Required/String) The name of the field to listen to and respond to changes in.
tableName: (Optional) Specifies the table in which the listened-to-field resides. Defaults to the same table as this field definition resides in.
recordKey: (Optional)
If record key level grouping is configured, specifies the record key of the field to listen on. This can be used in conjunction with tableName to target a very specific field that this field definition needs to respond to.
If omitted, this will default to the sub table that this field definition resides on.
An optional wildcard value of "ALL" can be specified which will listen to all subtables for the table associated with this field definition.
handlers: (Required/Array of handler configurations) The configurations which define which changes to respond to and how to respond to such changes
Handler Configurations
values: (Required/Array of strings) The values to attempt to match on; can have explicit values or can be an empty array to match arbitrary input.
match: (Optional - Defaults to "EQUALS") Determines how the above values are to be interpreted for the specified action to be taken. This field is a string constant, options:
EQUALS - matches the exact values for either the array or single value
NOT_EQUALS
IS_NOT_BLANK - matches any (non-blank) value
EQUALS_ONE_OF - matches any of the values in the array
NOT_EQUALS_ANY_OF - does not match any of the values in the array
success(Optional) The behavior this field will exhibit if the value expression is matched. This field is a string with possible values ("ENABLE" | "DISABLE" | "SHOW" | "HIDE")
failure(Optional) The behavior this field will exhibit if the value expression is not matched. This field is a string constant ("ENABLE" | "DISABLE" | "SHOW" | "HIDE")
It is important to note the difference between the following paired settings in the handlers section:
ENABLE/DISABLE: These settings control whether a field is enabled to accept user input based on the success or failure of the condition.
SHOW/HIDE: These settings control whether a field is show to the user based on the success or failure of the condition.
Specifically, if a field is initially hidden, your success handler must SHOW it; enabling a hidden field will not show the field to the user.
Conditional Logic Examples
The following examples will help illustrate using conditional logic.Scenario 1: You have two fields, "Metastasis" and "Cancer Recurrance", and only want to show the latter if the former is true, you would define the "Cancer Recurrance" field as:
Scenario 2: A field called diseaseGroup is a multi-select type. You want to enable a field only if the selections : "brain", "lung", and "breast" are selected and disabled otherwise. Within the definition of that field, the listeners portion would read:
Scenario 3 : You want a field to be shown only if any selection is made on the multi select field "testsCompleted". Within the field definition, the listeners section would look like:
"listeners" : [{ "field" : "testCompleted", "handlers" : [{ "values" : [], "match" : "IS_NOT_BLANK", "success" : "SHOW", "failure" : "HIDE" // will hide if there is no selection } ] }]]
Premium Feature — Available in the Enterprise Edition of LabKey Server. Learn more or contact LabKey.
The Document Abstraction (or Annotation) Workflow supports the movement and tracking of documents through the following general process. All steps are optional for any given document and each part of the workflow may be configured to suit your specific needs:
Workflow
Document Upload: with or without initial automatic abstraction to obtain some metadata and text fields.
Assignment to a Manual Abstractor and/or Reviewer: may be done automatically or manually.
Potential Reprocessing or Additional Abstraction Rounds
Approval
Possible Second or Subsequent Rounds of Review
Different types of documents (for example, Pathology Reports and Cytogenetics Reports) can be processed through the same workflow, task list and assignment process, each using abstraction algorithms specific to the type of document. The assignment process itself can also be customized based on the type of disease discussed in the document.
Manage project groups corresponding to the expected disease groups and document types.
Create document processing configurations.
Abstractor:
Choose a document to abstract from individual list of assignments.
Abstract document. You can save and return to continue work in progress if needed.
Submit abstraction for review - or approval if no reviewer is assigned.
Reviewer:
Review list of documents ready for review.
Review abstraction results; compare results from multiple abstractors if provided.
Mark document as ready to progress to the next stage - either approve or reject.
Review and potentially edit previously approved abstraction results.
It is important to note that documents to be abstracted may well contain protected health information (PHI). Protection of PHI is strictly managed by LabKey Server, and with the addition of the nlp_premium, compliance, and complianceActivites modules, all access to documents, task lists, etc, containing PHI can be gated by permissions and also subject to approval of terms of use specific to the user's intended activity. Further, all access that is granted, including viewing, abstracting, and reviewing can be logged for audit or other review.All sample screenshots and information shown in this documentation are fictitious.
Configuring Terminology
The terminology used in the abstraction user interface can be customized to suit the specific words used in the implementation. Even the word "Abstraction" can be replaced with something your users are used to, such as "Annotation" or "Extraction." Changes to NLP terminology apply to your entire site.An Abstraction Administrator can configure the terminology used as follows:
Select > Site > Admin Console.
Under Premium Features, click NLP Labels.
The default labels for concepts, fields, tooltips, and web part customization are listed.
There may be additional customizable terms shown, depending on your implementation.
Hover over any icon for more information about any term.
Enter new terminology as needed, then click Save.
To revert all terms to the original defaults, click Reset.
Terms you can customize:
The concept of tagging or extracting tabular data from free text documents (examples: abstraction, annotation) and name for user(s) who do this work.
Subject identifier (examples: MRN, Patient ID)
What is the item being abstracted (examples: document, case, report)
The document type, source, identifier, activity
The new terms you configure will be used in web parts, data regions, and UIs; in some cases admins will still see the "default" names, such as when adding new web parts.
To facilitate helping your users, you can also customize the tooltips that will be shown for the fields to use your local terminology.
You can also customize the subtable name used for storing abstraction information using a module property. See below
Abstraction Workflow
Documents are first uploaded, then assigned, then pass through the many options in the abstraction workflow until completion.The document itself passes through a series of states within the process:
Ready for assignment: when automatic abstraction is complete, automatic assignment was not completed, or reviewer requests re-abstraction
Ready for manual abstraction: once an abstractor is assigned
Ready for review: when abstraction is complete, if a reviewer is assigned
(optional) Ready for reprocessing: if requested by the reviewer
Approved
If Additional Review is requested, an approved abstraction result is sent for secondary review to a new reviewer.
Passage of a document through these stages can be done using a BPMN (business process management) workflow engine. LabKey Server uses an Activiti Workflow to automatically advance the document to the correct state upon completion of the prior state. Users assigned as abstractors and reviewers can see lists of tasks assigned to them and mark them as completed when done.
Assignment
Following the upload of the document and any provided metadata or automatic abstraction results, many documents will also be assigned for manual abstraction. The manual abstractor begins with any information garnered automatically and validates, corrects, and adds additional information to the abstracted results.The assignment of documents to individual abstractors may be done automatically or manually by an administrator. An administrator can also choose to bypass the abstraction step by unassigning the manual abstractor, immediately forwarding the document to the review phase.
The Abstraction Task List web part is typically included in the dashboard for any NLP project, and shows each viewing user a tailored view of the particular tasks they are to complete. Typically a user will have only one type of task to perform, but if they play different roles, such as for different document types, they will see multiple lists.
Abstraction
The assigned user completes a manual document abstraction following the steps outlined here:
Once abstraction is complete, the document is "ready for review" (if a reviewer is assigned) and the task moves to the assigned reviewer. If the administrator chooses to bypass the review step, they can leave the reviewer task unassigned for that document.
Reviewers select their tasks from their personalized task list, but can also see other cases on the All Tasks list. In addition to reviewing new abstractions, they can review and potentially reject previously approved abstraction results. Abstraction administrators may also perform this second level review. A rejected document is returned for additional steps as described in the table here.
Setting Module Properties
Module properties are provided for customizing the behavior of the abstraction workflow process, particularly the experience for abstraction reviewers. Module properties, also used in configuring NLP engines, can have both a site default setting and an overriding project level setting as needed.
ShowAllResults: Check the box to show all sets of selected values to reviewers. When unchecked, only the first set of results are shown.
AnonymousMode: Check to show reviewers the column of abstracted information without the userID of the individual abstractor. When unchecked, the name of the abstractor is shown.
Default Record Key: The default grouping name to use in the UI for signaling which record (e.g. specimen or finding) the field value relates to. Provide the name your users will expect to give to subtables. Only applicable when the groupings level is at 'recordkey'. If no value is specified, the default is "SpecimenA" but other values like "1" might be used instead.
To set module properties, select > Folder > Management and click the Module Properties tab.
Scroll down to see and set the properties relevant to the abstraction review process:
Click Save Changes.
Developer Note: Retrieving Approved Data via API
The client API can be used to retrieve information about imported documents and results. However, the task status is not stored directly, rather it is calculated at render time when displaying task status. When querying to select the "status" of a document, such as "Ready For Review" or "Approved," the reportId must be provided in addition to the taskKey. For example, a query like the following will return the expected calculated status value:
SELECT reportId, taskKey FROM Report WHERE ReportId = [remainder of the query]
When setting up automatic task assignment, the abstraction administrator defines named configurations for the different types of documents to be abstracted and different disease groups those documents cover. The administrator can also create specific project groups of area experts for these documents so that automatic assignment can draw from the appropriate pool of people.
Project Group Curation
The abstraction administrator uses project groups to identify the people who should be assigned to abstract the particular documents expected. It might be sufficient to simply create a general "Abstractors" group, or perhaps more specific groups might be appropriate, each with a unique set of members:
Lung Abstractors
Multiple Myeloma Abstractors
Brain Abstractors
Thoracic Abstractors
When creating document processing configurations, you can select one or more groups from which to pull assignees for abstraction and review.
Create the groups you expect to need via > Folder > Permissions > Project Groups.
On the Permissions tab, add the groups to the relevant abstraction permission role:
Abstractor groups: add to Document Abstractor.
Reviewer groups: add to Abstraction Reviewer.
Neither of these abstraction-specific roles carries any other permission to read or edit information in the folder. All abstractors and reviewers will also require the Editor role in the project in order to record information. Unless you have already granted such access to your pool of users through other groups, also add each abstractor and reviewer group to the the Editor role.
Next add the appropriate users to each of the groups.
While the same person may be eligible to both abstract some documents and review others, no document will be reviewed by the same person who did the abstraction.
NLP Document Processing Configurations
Named task assignment configurations are created by an administrator using an NLP Document Processing Configurations web part. Configurations include the following fields:
Name
DocumentType
Pathology Reports
Cytogenetics Reports
Clinical Reports
All Documents (including the above)
Disease Groups - check one or more of the disease groups listed. Available disease groups are configured via a metadata file. The disease group control for a document is generated during the initial processing during upload. Select "All" to define a configuration that will apply to any disease group not covered by a more specific configuration.
ManualAbstractPct - the percentage of documents to assign for manual abstraction (default is 5%).
ManualAbstractReviewPct - the percentage of manually abstracted documents to assign for review (default is 5%).
EngineAbstractReviewPct - the percentage of automatically abstracted documents to assign for review (default is 100%).
MinConfidenceLevelPct - the minimum confidence level required from an upstream NLP engine to skip review of those engine results (default is 75%).
Assignee - use checkboxes to choose the group(s) from which abstractors should be chosen for this document and disease type.
Status - can be "active" or "inactive"
Other fields are tracked internally and can provide additional information to assist in assigning abstractors:
DocumentsProcessed
LastAbstractor
LastReviewer
You can define different configurations for different document types and different disease groups. For instance, standard pathology reports might be less likely to need manual abstraction than cytogenetics reports, but more likely to need review of automated abstraction. Reports about brain diseases might be more likely to need manual abstraction than those about lung diseases. The document type "All Documents" and the disease group "All" are used for processing of any documents not covered by a more specific configuration. If there is a type-specific configuration defined and active for a given document type, it will take precedence over the "All Documents" configuration. When you are defining a new configuration, you will see a message if it will override an existing configuration for a given type.You can also define multiple configurations for a given document type. For example, you could have a configuration requiring higher levels of review and only activate it during a training period for a new abstractor. By selecting which configuration is active at any given time for each document type, different types of documents can get different patterns of assignment for abstraction. If no configuration is active, all assignments must be done manually.
Outcomes of Automatic Document Assignment
The following table lists what the resulting status for a document will be for all the possible combinations of whether engine abstraction is performed and whether abstractors or reviewers are assigned.
Engine Abstraction?
Abstractor Auto-Assigned?
Reviewer Auto-Assigned?
Document Status Outcome
Y
Y
Y
Ready for initial abstraction; to reviewer when complete
Y
Y
N
Ready for initial abstraction; straight to approved when complete
Y
N
Y
Ready for review (a common case when testing engine algorithms)
Y
N
N
Ready for manual assignment
N
Y
Y
Ready for initial abstraction; to reviewer when complete
N
Y
N
Ready for initial abstraction; straight to approved when complete
Premium Feature — Available in the Enterprise Edition of LabKey Server. Learn more or contact LabKey.
When documents need to be manually assigned to an abstractor, they appear as tasks for an abstraction administrator. Assigned tasks in progress can also be reassigned using the same process. This topic describes the steps to make manual assignments of abstraction tasks and batches.
The task list view allows manual assignment of abstractors and reviewers for a given document. To be able to make manual assignments, the user must have "Abstraction Administrator" permission; folder and project administrators also have this permission.Users with the correct roles are eligible to be assignees:
Abstractors: must have both "Document Abstractor" and "Editor" roles.
Reviewers: must have both "Abstraction Reviewer" and "Editor" roles.
It is good practice to create project groups of eligible assignees and granted the appropriate roles to these groups, as described here.Each user assigned to an abstraction role can see tasks assigned to them and work through a personalized task list.Click Assign on the task list.In the popup, the pulldowns will offer the list of users granted the permission necessary to be either abstrators or reviewers. Select to assign one or both tasks. Leaving either pulldown without a selection means that step will be skipped.
Click Save and the document will disappear from your "to assign" list and move to the pending task list of the next user you assigned.
Reassignment and Unassignment
After assigment, the task is listed in the All Cases grid. Here the Assign link allows an administrator to change an abstraction or review assignment to another person.If abstraction has not yet begun (i.e. the document is still in the "Ready for initial abstraction" state), the administrator can also unassign abstraction by selecting the null row on the assignment pulldown. Doing so will immediately send the document to the review step, or if no reviewer is assigned, the document will be approved and sent on.Once abstraction has begun, the unassign option is no longer available.
Batch Assignment and Reassignment
Several document abstraction or review tasks can be assigned or reassigned simultaneously as a batch, regardless of whether they were uploaded as part of the same "batch". Only tasks which would be individually assignable can be included in a batch reassignment. If the task has an Assign link, it can be included in this process:
From the Abstraction Task List, check the checkboxes for the tasks (rows) you want to assign to the same abstractor or reviewer (or both).
Click Assign in the grid header bar.
Select an Initial Abstractor or Reviewer or both.
If you leave the default "No Changes" in either field, the selected tasks will retain prior settings for that field.
If you select the null/empty value, the task will be unassigned. Unassignment is not available once the initial abstraction is in progress. Attempting to unassign an ineligible document will raise a warning message that some of the selected documents cannot be modified, and you will have the option to update all other selected documents with the unassignment.
Click Save to apply your changes to all tasks.
Newly assigned abstractors and reviewers will receive any work in progress and see previously selected values when they next view the assigned task.
Administrators, abstractors, and reviewers have a variety of options for displaying and organizing tasks in the Natural Language Processing (NLP) abstraction process workflow.
If your folder does not already have the Abstraction Task List or NLP Batch View web parts, an administrator can add them.
Abstraction Task List
The Abstraction Task List web part will be unique for each user, showing a tailored view of the particular tasks they are to complete, including assignment, abstraction, and review of documents.Typically a user will have only one type of task to perform, but if they play different roles, such as for different document types, they will see multiple lists. Tasks may be grouped in batches, such as by identifier or priority, making it easier to work and communicate efficiently. Below the personalized task list(s), the All Cases list gives an overview of the latest status of all cases visible to the user in this container - both those in progress and those whose results have been approved. In this screenshot, an user has both abstract and review tasks. Hover over any row to reveal a (Details) icon link for more information.All task lists can be sorted to provide the most useful ordering to the individual user. Save the desired sorted grid as the "default" view to use it for automatically ordering your tasks. When an abstraction or review task is completed, the user will advance to the next task on their default view of the appropriate task list.
Task Group Identifiers (Activities)
Abstraction cases can be assigned an identifier, allowing administrators to group documents for abstraction and filter based on something specific about the activity or category of document.
You can change the label used in the user interface to something else, such as Activity if that better matches your existing workflows. The schema browser and some parts of the UI may still use the word "Identifier" but grid and field labels will read "Activity".
Identifiers correspond to batches or groups of tasks, and are used to organize information in various ways. To define identifiers to use in a given folder, use the schema browser.
Select > Go To Module > Query
Open the nlp schema.
Select Identifier from the built in queries and tables.
Click View Data.
To add a new identifier, click (Insert data) > Insert new row.
Enter the Name and Description.
Enter the Target Count of documents for this identifier/activity.
Click Submit.
When cases are uploaded, the pipeline protocol determines which identifier/activity they are assigned to. All documents imported in a batch are assigned to the same identifier/activity.
NLP Batch View
The Batch View web part can help abstractors and reviewers organize tasks by batch and identifier. It shows a summary of information about all cases in progress, shown by batch. Button options include the standard grid buttons as well as:
Change Identifier(or "Change Activity" if you have changed the terminology)
Extra Review: If additional rounds of review are requested, documents with approved results will show an Assign link in this column, allowing administrators to assign for the additional review step.
Identifier Name: Using identifiers allows administrators to group related batches together. Learn to change identifiers below.
Document Type
Dataset: Links in this column let the user download structured abstraction results for batches. Details are below.
Schema: The metadata for this batch, in the form of a .json file.
# documents: Total number of documents in this batch.
# abstracted: The number of documents that have gone through at least one cycle of abstraction.
# reviewed: The number of documents that have been reviewed at least once.
Batch Summary: A count of documents in each state.
Job Run ID
Input File Name
Customize Batch View
Administrators can customize the batch view to show either, both, or neither of the Dataset and Schema columns, depending on the needs of your users. By default, both are shown.
Open the (triangle) menu in the corner of the webpart.
Select Customize.
Check the box to hide either or both of the Dataset and Schema columns.
Change Identifier/Activity
Administrators viewing the Batch View web part will see a button for changing identifiers/activities associated with the batch. Select one or more rows using the checkboxes, then click Change Identifier/Activity to assign a new one to the batch.
Assign for Secondary Review
If additional rounds of review are requested as part of the workflow, the administrator can use the Batch View web part to assign documents from a completed batch for extra rounds of review.
If an NLP Batch View web part does not exist, create one.
In the Extra Review column, you will see one of the following values for each batch:
"Batch in progress": This batch is not eligible for additional review until all documents have been completed.
"Not assigned": This batch is completed and eligible for assignment for additional review, but has not yet been assigned.
"Assigned": This batch has been assigned and is awaiting secondary review to be completed.
Click the Assign link for any eligible batch to assign or reassign.
In the popup, use checkmarks to identify the groups eligible to perform the secondary review. You can create distinct project groups for this purpose in cases where specific teams need to perform secondary reviews.
Enter the percent of documents from this batch to assign for secondary review.
Click Submit.
The selected percentage of documents will be assigned to secondary reviewers from the selected groups.
The assigned secondary reviewers will now see the documents in their task list and perform review as in the first review round, with the exception that sending documents directly for reprocessing from the UI is no longer an option.
Dataset Download
Links in the Dataset column of the batch view let the user download structured abstraction results for batches. The name of the link may be the name of the originally uploaded file, or the batch number.These downloads are audited as "logged select query events".The exported zip contains a json file and a collection of .txt files. The json file provides batch level metadata such as "diseaseGroup", "documentType", "batchID" and contains all annotations for all reports in the batch.
Premium Feature — Available in the Enterprise Edition of LabKey Server. Learn more or contact LabKey.
Abstraction of information from clinical documents into tabular data can unearth a wealth of previously untapped data for integration and analysis, provided it is done efficiently and accurately. An NLP engine can automatically abstract information based on the type of document, and additional manual abstraction by one or more people using the process covered here can maximize information extraction.
Tasks have both a Report ID and a Batch Number, as well as an optional Identifier Name, all shown in the task list. The assigned user must have "Abstractor" permissions and will initiate a manual abstraction by clicking Abstract on any row in their task list.The task list grid can be sorted and filtered as desired, and grid views saved for future use. After completion of a manual abstraction, the user will advance to the next document in the user's default view of the task list.
Abstraction UI Basics
The document abstraction UI is shown in two panels. Above the panels, the report number is prominently shown.The imported text is shown on the right and can be scrolled and reviewed for key information. The left hand panel shows field entry panels into which information found in the text will be abstracted.One set of subtables for results, shown in the above screenshot named "Specimen A", is made available. You can add more using the "Add another specimen" option described below. An admininstrator can also control the default name for the first subtable here. For instance, it could be "1" instead of "Specimen A".The abstracted fields are organized in categories that may vary based on the document type. For example, pathology documents might have categories as shown above: Pathology, PathologyStageGrade, EngineReportInfo, PathologyFinding, NodePathFindingExpand and contract field category sections by clicking the title bars or / icons. By default, the first category is expanded when the abstractor first opens the UI.
Fields in the "Pathology" category include:
PathHistology
PathSpecimenType
Behavior
PathSite
Pathologist
If an automated abstraction pass was done prior to manual abstraction, pulldowns may be prepopulated with some information gathered by the abstraction (NLP) engine. In addition, if the disease group has been automatically identified, this can focus the set of values for each field offered to a manual abstractor. The type of document also drives some decisions about how to interpret parts of the text.
Populating Fields
Select a field by clicking the label; the selected row will show in yellow, as will any associated text highlights previously added for that field. Some fields allow free text entry, other fields use pulldowns offering a set of possible values.Start typing in either type of field to narrow the menu of options, or keep typing to enter free text as appropriate. There are two types of fields with pulldown menus.
Open-class fields allow you to either select a listed value or enter a new value of your own.
Closed-class fields (marked with a ) require a selection of one of the values listed on the pulldown.
Multiple Value Fields
Fields supporting multiple entries allow you to click several pulldown menu using the shift or ctrl key when you click to add the new value instead of replacing the prior choice. The values show with a || (double pipe) separating them in the field value.
Conditional or Branching Logic Fields
When an abstractor selects a value for a field supporting conditional or branching logic, it may trigger additional fields or sections to be shown or hidden. This behavior helps the abstractor focus on what is relevant as they determine more about the case report.For example, if the abstractor sees that an ALK test was reported, they can enter the method and result, but if the test was not reported, those fields remain grayed out and inaccessible.In other cases, the set of fields offered may change outright based on the abstractor's selections for other fields. Learn more about how administrators configure such fields in this topic
Highlighting Text
The abstractor scans for relevant details in the text, selects or enters information in the field in the results section, and can highlight one or more relevant pieces of text on the right to accompany it.If you highlight a string of text before entering a value for the active field, the selected text will be entered as the value if possible. For a free text field, the entry is automatic. For a field with a pulldown menu, if you highlight a string in the text that matches a value on the given menu, it will be selected. If you had previously entered a different value, however, that earlier selection takes precedence and is not superceded by later text highlighting. You may multi-select several regions of text for any given field result as needed.In the following screenshot, several types of text highlighting are shown. When you click to select a field, the field and any associated highlights are colored yellow. If you double-click the field label, the text panel will be scrolled to place the first highlighted region within the visible window, typically three rows from the top. Shown selected here, the text "Positive for malignancy" was just linked to the active field Behavior with the value "Malignant". Also shown here, when you hover over the label or value for a field which is not active, in this case "PathHistology" the associated highlighted region(s) of text will be shown in green.Text that has been highlighted for a field that is neither active (yellow) nor hovered-over (green) is shown in light blue. Click on any highlighting to activate the associated field and show both in yellow.A given region of text can also be associated with multiple field results. The count of related fields is shown with the highlight region ("1 of 2", for example).Unsaved changes are indicated by red corners on the entered fields. If you make a mistake or wish to remove highlighting on the right, click the 'x' attached to the highlight region.
Save Abstraction Work
Save work in progress any time by clicking Save Draft. If you leave the abstraction UI, you will still see the document as a task waiting to be completed, and see the message "Initial abstraction in progress". When you return to an abstraction in progress, you will see previous highlighting, selections, and can continue to review and abstract more of the document.Once you have completed the abstraction of the entire document, you will click Submit to close your task and pass the document on for review, or if no review is selected, the document will be considered completed and approved.When you submit the document, you will automatically advance to the next document assigned for you to abstract, according to the sort order established on your default view of your task list. There is no need to return to your task list explicitly to advance to the next task. The document you just completed will be shown as "Previously viewed" above the panels.If you mistakenly submit abstraction results for a document too quickly, you can use the back button in your browser to return. Click Reopen to return it to an "abstraction in progress" status.
Abstraction Timer
If enabled, you can track metrics for how much time is spent actively abstracting the document, and separately time spent actively reviewing that abstraction. The timer is enabled/disabled based on which pipeline is used.The abstraction timer is displayed above the document title and automatically starts when the assigned abstractor lands on the page. If the abstractor needs to step away or work on another task, they may pause the timer, then resume when they resume work on the document. As soon as the abstractor submits the document, the abstraction timer stops.The reviewer's time on the same document is also tracked, beginning from zero. The total time spent in process for the document is the sum of these two times.Note that the timer does not run when others, such as administrators, are viewing the document. It only applies to the edit-mode of active abstracting and reviewing.
Session Timeout Suspension
When abstracting a particularly lengthy or complicated document, or one requiring input from others, it is possible for a long period of time to elapse between interactions with the server. This could potentially result in the user’s session expiring, especially problematic as it can result in the loss of the values and highlights entered since the last save. To avoid this problem, the abstraction session in progress will keep itself alive by pinging the server with a lightweight "keepalive" function while the user continues interacting with the UI.
Multiple Specimens per Document
There may be information about multiple specimens in a single document. Each field results category can have multiple panels of fields, one for each specimen. You can also think of these groupings as subtables, and the default name for the first/default subtable can be configured by an administrator at the folder level. To add information for an additional specimen, open the relevant category in the field results panel, then click Add another specimen and select New Specimen from the menu.Once you have defined multiple specimen groupings for the document, you will see a panel for each specimen into which values can be entered.Specimen names can be changed and specimens deleted from the abstraction using the cog icon for each specimen panel.
Reopen an Abstraction Task
If you mistakenly submit abstraction results for a document too quickly, you can use the back button in your browser to return to the document. Click Reopen to return it to an unapproved status.
Premium Feature — Available in the Enterprise Edition of LabKey Server. Learn more or contact LabKey.
Once document abstraction is complete, if a reviewer is assigned to the document, the status becomes "ready for review" and the task moves to the assigned reviewer. If no reviewer is assigned, the document abstraction will bypass the review step and the status will be "approved."
This topic covers the review process when there is a single result set per document. When multiple result sets exist, a reviewer can compare and select among the abstractions of different reviewers following this topic: Review Multiple Result Sets.
Single Result Set Review
The review page shows the abstracted information and source text side by side. Only populated field results are displayed by default. Hover over any field to highlight the linked text in green. Click to scroll the document to show the highlighted element within the visible window, typically three rows from the top. A tooltip shows the position of the information in the document.
Edit Result Set
To see all available fields, and enable editing of any entries or adding any additional abstraction information, the reviewer can click the pencil icon at the top of the field results panel.In this mode, the reviewer can edit the abstractors entered text or select new values from drop down menus as appropriate. See Document Abstraction for details. Note that if any fields are configured to use conditional or branching logic, when a reviewer changes an entry, the set of available fields may also change requiring additional field selections in some cases.Once the pencil icon has opened the abstraction results for potential editing, the reviewer has the intermediate option to Save Draft in order to preserve work in progress and return later to complete their review.
Complete Review
The reviewer finishes with one of the following clicks:
Approve to accept the abstraction and submit the results as complete. If you mistakenly click approve, use your browser back button to return to the open document; there will be a Reopen button allowing you to undo the mistaken approval.
Reprocess which rejects the abstraction results and returns the document for another round of abstraction. Either the engine will reprocess the document, or an administrator will assign a new manual abstractor and reviewer.
If you select Reprocess, you will be prompted to enter the cause of rejection.After completing the review, you will immediately be taken to the next document in your default view of your review task list.
Reprocessing
When a reviewer clicks Reprocess, the document will be given a new status and returned for reprocessing according to the following table:
Engine Abstracted?
Manually Abstracted?
Reviewed?
Action
Result
Yes
No
No
Reopen
Ready for assignment
Yes
No
Yes
Reopen
Ready for review; assign to same reviewer
Yes
No
Yes
Reprocess
Engine reprocess; ready for assignment
Yes
Yes
No
Reopen
Ready for assignment
Yes
Yes
Yes
Reopen
Ready for review; assign to same reviewer
Yes
Yes
Yes
Reprocess
Engine reprocess, then ready for assignment
No
Yes
No
Reopen
Ready for assignment
No
Yes
Yes
Reopen
Ready for review; assign to same reviewer
No
Yes
Yes
Reprocess
Ready for assignment
Reopen is an option available to administrators for all previously approved documents. Reviewers are only able to reopen the documents they reviewed and approved themselves.
Premium Feature — Available in the Enterprise Edition of LabKey Server. Learn more or contact LabKey.
Document abstraction by multiple abstractors may generate multiple result sets for the same document. This topic describes how reviewers can compare multiple, possibly differing, abstracted values and select the correct values for each field. If you only have one result set to review, use this topic instead: Review Document Abstraction.
Select the document for review from the task list.
View Multiple Result Sets
If there are multiple abstraction result sets, they will be shown in columns with colored highlighting to differentiate which abstractor chose which result. Each abstractor is assigned a color, shown in a swatch next to the user name. Colors are hard coded and used in this order:
Abstractor 1
Abstractor 2
Abstractor 3
Abstractor 4
Abstractor 5
In the following screenshot, you see two abstraction results sets for the same document were created. Values where the abstractors are in agreement are prepopulated, and you can still make adjustments if needed as when reviewing a single abstraction set.Only one abstractor chose a value for "PathQuality" and different selections were made for the "PathSite".
Anonymous Mode
An administrator can configure the folder to use anonymous mode. In this mode, no abstractor user names are displayed; instead, labels like "Abstractor 1", "Abstractor 2", etc. are used instead.
Compare Highlights
When you select a field, you will see highlights in the text panel, color coded by reviewer. When multiple abstractors agreed and chose the same text to highlight, the region will be shown striped with both abstractor colors.When the abstractors did not agree, no value will be prepopulated in the left panel. Select the field to see the highlights; in this case, the highlights still agree, and there is no shown indication of why the first abstractor chose mediastinum. As a reviewer, you will use the dropdown to select a value - values chosen by abstractors are listed first, but you could choose another value from the list if appropriate. In this case, you might elect to select the "Lung" value.While working on your review, you have the option to Save Draft and return to finish later. Once you have completed review of the document, click Approve or Reprocess if necessary. The completion options for review are the same as when reviewing single result sets.
Review Conditional or Branching Logic Selections
When fields are configured as conditional, it's possible that different annotators will make different selections and trigger different sets of fields.In this screencap, you can see a number of examples showing the notations and field results. Crosshatching indicates a field that was not presented to the annotator. A "(No value)" notation indicates that the annotator was shown the field but did not choose a result.
1. For the "ALK Test Reported" field, two annotators chose yes, one chose no, and one did not select any value. You can see that the two following fields, "ALK Test Method" and "ALK Test Result" were conditional, and not even available to annotators 2 and 3. They will remain grayed out until the reviewer makes a decision about which answer to accept for "ALK Test Reported".
2. The "EGFR Test Reported" field was unanimously agreed to be "Yes" and so is populated on the left and the conditional fields shown below it, but without values until the reviewer assigns them.
3. A secondary conditional field, "EGFR Positive Test Results" was only seen by Annotator 3, who said the "EGFR Test Result" was positive. Only if the reviewer agrees with that annotator about the first two fields will they need to select a value for the field.
As with review of any single result, the reviewer is empowered to select a new value for any field based on their own review of the original report and results. The same branching logic for any conditional fields will be applied. If, for example, they changed the entire disease group, they might see an entirely new set of fields to populate.
Configure Settings
In the lower left corner, click Settings to configure the following options.
Show highlight offsets for fields: When you hover over a value selected by an abstractor, you see the full text they entered. If you want to instead see the offsets for the highlights they made, check this box.
Enable comparison view: Uncheck to return to the single result set review display. Only results selected by the first abstractor are shown.
Premium Feature — Available in the Enterprise Edition of LabKey Server. Learn more or contact LabKey.
This topic describes transferring Natural Language Processing (NLP) Result Sets, also referred as abstraction or annotation results, from one server to another. For example, a number of individual Registry Servers could submit approved results to a central Data Sharing Server for cross-registry comparison and review.
A user creates an API key on the destination server. This key is used to authorize data uploads sent from the source server, as if they came from the user who generated the API key. Any user with write access to the destination container can generate a key, once an administrator has enabled this feature.On the destination server:
Select (Your_Username) > API Keys.
Note: Generation of API Keys must be enabled on your server to see this option. Learn more: API Keys.
Click Generate API Key.
Click Copy to Clipboard to do so and share with the system administrator of the source server in a secure manner.
On the source server, an administrator performs these steps:
Select > Site > Admin Console.
Under Premium Features, click NLP Transfer.
Enter the required information:
API Key: Enter the key provided by the destination server/recipient.
Encryption Passphrase: The passphrase to use for encrypting the results, set above.
Base URL: The base URL of the destination server. You will enter the path to the destination folder relative to this base in the module properties below.
Click Test Transfer to confirm a working connection.
Click Save.
Module Properties
In each folder on the source server containing results which will be exported, a folder administrator must configure these properties.
Select > Folder > Management.
Click the Module Properties tab.
Scroll down to Property: Target Transfer Folder.
Enter the path to the target folder relative to the Base URL on the destination server, which was set in the NLP Transfer configuration above.
Note that you can have a site default for this destination, a project-wide override, or a destination specific to this individual folder.
Transfer Results to Destination
To transfer data to the destination, begin on the source server:
Navigate to the Batch View web part containing the data to transfer.
Three buttons related to transferring results are included.
Export Results: Click to download an unencrypted bundle of the selected batch results. Use this option to check the format and contents of what you will transfer.
Transfer Data: Click to bundle and encrypt the batches selected, then transfer to the destination. If this button appears inactive (grayed out), you have not configured all the necessary elements.
Config Transfer: click to edit the module property to point to the correct folder on the destination server if not done already.
Select the desired row(s) and click Transfer Data.
Note: The Batch View web part shows both Dataset and Schema columns by default. Either or both of those columns may have been hidden by an administrator from the displayed grid but both contain useful information that is transferred.
Zip File Contents and Process
Result set files, with files in JSON, XML, and TSV format. The entire bundle is encrypted using the encryption key entered on the source server. Results can only be decrypted using this password. The encrypted file is identical to that which would be created with this command line:
Statistics: # of documents, # abstracted, #reviewed
Use Results at the Destination
The destination server will receive the transfer in the folder configured from the source server above.Developers can access the results using the schema browser, (schema = "nlp", query "ResultsArchive", click "View Data"). For convenience, an admin can create a web part to add it to a folder as follows:
Navigate to the folder where results were delivered.
Enter > Page Admin Mode.
Select Query from the selector in the lower left, then click Add.
Give the web part a title, such as "Results Archive".
Select the Schema: nlp.
Click Show the contents of a specific query and view.
Select the Query: ResultsArchive.
Leave the remaining options at their defaults.
Click Submit.
Users viewing the results archive will see a new row for each batch that was transferred.The columns show:
Source: The name of the source server; shown here regional registries.
Schema: The name of the metadata.json file.
Dataset: A link to download the bundled and encrypted package of results transferred.
Document Type, # reports, # abstracted, and # reviewed: Information provided by the source server.
Created/Created By: The transfer time and username of the user who created the API key authorizing the upload.
To unpack the results:
Click the link in the Dataset column to download the transferred bundle.
Decrypt it using this command:
gpg -o test.txt.zip -d test.txt.zip.gpg
Unzip the decrypted archive to find all three export types (JSON, TSV, XML) are included.
Folder editors and administrators have the ability to delete transferred archives. Deleting a row from the Results Archive grid also deletes the corresponding schema JSON and gpg (zip archive) files.
LabKey Server helps you model instrument data in a manageable and meaningful way. Organize large quantities of complex data, analyze integrated datasets, and prepare experiment data for integration with related clinical, demographic, and sample information.LabKey captures experimental data using the assay framework. An assay design tells LabKey Server how to interpret your instrument data and the context around it. All common tabular formats are supported, as well as a wide variety of instrument-specific formats.The experimental/assay framework also provides the following general features:
Handling large amounts of data, generated repeatedly and over time.
Tracking both data and metadata.
Cleaning, validation, transformation of assay/experimental data.
Integration with other types of related data, such as clinical data.
Publishing and sharing selected data in a secure way with collaborators.
Standard Assays
Standard assays are the most flexible choice for working with experimental data. For each assay design, you customize the format for mapping your experimental results.All common assay file types are supported:
Excel formats (.xls, .xlsx)
Tab or Comma Separated Values (.csv, .tsv)
Text files
Protein sequencing formats (.fna, .fasta)
In general, if your instrument provides tabular data, then the data can be imported using a Standard Assay. Standard assays are highly-flexible and configurable by the user, so you can extend them to fit your needs.
A developer can create a specialized "module-based" or "file-based" assay type that provides additional functionality. When created and deployed, these assay types will be available as Specialty Assays when you create a new assay design.
In Premium Editions, LabKey also offers many instrument-specific assay types, including NAb, ELISpot, ELISA, Luminex, etc. built in to LabKey Server. See the full list below and contact your Account Manager if you would like to add any of these assays to your premium edition.These Specialty Assays pre-configure commonly used structures and metadata fields, and many support special built-in dashboards and reports. Specialty assays are also highly-flexible and configurable by the user, so you can extend the reach of any available type to fit your needs.
Specialty Assay Types
LabKey assay tools simplify complex laboratory workflows and incorporate custom analytics and quality control and trending tools for specific instrument types. Supported types include:
ELISA - Imports raw data from BioTek Microplate readers.
ELISpot - Imports raw data files from CTL, Zeiss, and AID instruments.
Fluorospot - Similar to ELISpot, but the detection mechanism uses fluorophores instead of enzymes.
Luminex - Imports data in the multi-sheet BioPlex Excel file format.
This tutorial walks you through capturing, importing, and interpreting a spreadsheet of experimental results. In particular, this tutorial will show how to:
Capture the experiment data in LabKey Server using a new Assay Design.
Import the data to the server.
Visualize the data.
As part of this tutorial, we will create an Assay Design, a data structure for the import and storage of experiment data. The Assay Design is based on the General assay type, the most flexible of LabKey Server's tools for working with experiment data. This assay type gives you complete control over the format for mapping your experimental results into LabKey Server.The tutorial provides example data in the form of a Cell Culture assay that compares the performance of different media recipes.
Step 2: Infer an Assay Design from Spreadsheet Data
When you import assay data into LabKey server, the process is handled by an Assay Design, which tells the server how to interpret the information in your data files. If you don't yet have an Assay Design that matches your file structure, you can create one manually or you can infer one automatically from one of your data files. In the step below, we use the quicker method of inferring column names and types from one of our example data files.
Create a New Assay Design by Inference
In the Files web part, check the box for "CellCulture_001.xls".
Click Import Data.
In the Import Data popup, locate the Text or Excel Assay section, select Create New Standard Assay Design and click Import.
This will kick off the inference wizard.
Enter the Name: "Cell Culture"
You can click the Fields sections to see what was inferred and make changes if needed. For this tutorial, accept the inferrals.
When creating your own assay designs by inferral, use the guidance in this section, particularly if your data fields contain any special characters (including spaces). Using "database-legal" names without special characters is highly recommended. You can use labels to show users column names including special characters.
Click Save.
The Assay Design has now been created and saved.
You are directed to the first page of the data import wizard.
On the "Data Import: Batch Properties" page, click Cancel.
This quits the data import wizard without importing the file you used for inferral. We will import all four files in one batch in the next step of the tutorial.
Click Assay Tutorial near the top of the page to return to the main folder page. You will now see Cell Culture on the assay list.
Once we have created an assay design, we can use it to import many files of the same data structure into the assay framework it describes. In the step below, we will import all four assay files as one batch into the Cell Culture assay design we just created.
Import Multiple Files into the Assay Design
If necessary, return to the main page by clicking the Assay Tutorial link near the top of the page.
In the Files web part, check the boxes for the four files, each of which represents a single run.
CellCulture_001.xls
CellCulture_002.xls
CellCulture_003.xls
CellCulture_004.xls
Click Import Data.
In the pop up dialog, select Use Cell Culture (if necessary -- it may already be selected by default). This is the assay design you just created.
Click Import.
This will queue up all four files for import into the Assay Design.
You will be navigated to the assay data import wizard. The wizard proceeds through different phases: first collecting properties that you want to attach to the whole "batch" of files, and the remainder collecting properties on each individual file you are importing.
See the final step of the tutorial for more information on using Batch properties.
On the Batch Properties page, make no changes, and click Next.
The import wizard will now be on the "Run Properties and Data File" page, which captures any properties that you want to attach individually to the first file being imported (i.e., CellCulture_001.xls). See the final step of the tutorial for more information on using Run properties.
Notice the Batch Properties are shown in a read-only panel for reference.
Run Data: Notice the file being imported: "CellCulture_001.xls". The properties you just entered will be attached to this file.
On the Run Properties and Data File page, make no changes and click Save and Import Another Run.
Repeat for the next two files in the queue.
For the final file, click Save and Finish. (Notice the "Save and Import Another Run" button is gone.)
Once all of the files have been imported, you will be navigated to the Cell Culture Runs grid.
Once you have imported your assay data, you can analyze the results and generate visualizations of your data. This topic covers some ways to get started visualizing assay data.
Assay Results
The Results grid shows the assay data originally encoded in the Excel files, now captured by the Assay Design. In this step, we navigate to the results and create a visualization based on the data.
On the Cell Culture Runs grid, select all rows and click Show Results.
You are taken to the Cell Culture Results grid.
It will show a run number filter, which you can hide by clicking the 'X'.
Use the data grid to create custom filtered views and visualizations. Two example visualizations are shown below.
Column Visualizations
To add two visualizations to the top of the grid:
Click the column header Total Cell Count and select Box and Whisker.
Click the column header Media and select Pie Chart.
Media Performance Compared
To create a visualization of cell growth over time, we use a chart that combines data from several columns.
Above the grid, click (Charts/Reports) and select Create Chart.
In the popup dialog, click Line in the panel of chart types.
Drag-and-drop columns into the form as follows:
X Axis - Culture Day
Y Axis - Total Cell Count
Series - Media
Trendline - Leave the default "Point-to-Point" selected.
The following line chart is created, comparing the performance of the different media.
You can reopen the plot editor by clicking Chart Type.
You can create various other types of plots here if you like.
Experiment with trendline and using other fields for the axes and series if you like.
Experiment with changing the look and labeling of the chart by clicking Chart Layout.
When finished, you can either Save this chart for use in a dashboard
OR
Cancel by clicking Assay Tutorial to return to the main folder page.
In the next step, learn how to expand the assay design beyond the information in the data file itself.
In many experiments, relevant information is not contained in the experimental data file itself. This metadata or contextual data is often important in the downstream analysis of the experiment. For example, all of the factors below might be important for analyzing and evaluating an experiment, but often they are not included in the files generated by an assay instrument:
who ran the experiment
what lab processed the samples, and other provenance factors
the ambient temperature and humidity in the lab
analytes and reagents used
titration values
the experimental protocol used and modifications to the protocol
instrument settings and configuration
participant ids and visit numbers
project name and/or funding sources
etc.
LabKey Server lets you attach different kinds of metadata to experiment/assay data:
Batch Fields - These fields attach to groups of imported files as a whole. For example, you might attach a project name, "Project A", to a whole batch of files to group them together.
Run Fields - These fields are attached to individual files, or "runs", of data. For example, you might attach an instrument operator name or the instrument settings to an individual data file to be used as a quality control differentiator or as a factor in subsequent analyses.
In the step below, we will modify the Assay Design by adding a Run metadata field, which will capture the "Processing Lab" that generated the data.
Add a Run Metadata Field
Open the Assay Design editor:
Click Assay Tutorial to return to the main page.
In in the Assay List panel, click Cell Culture.
Select Manage Assay Design > Edit Assay Design.
If you defined the assay in the project, a popup dialog appears saying "This assay is defined in the /Tutorials folder. Would you still like to edit it?" Click OK to proceed to the editor.
In the General Assay Designer, the section Assay Properties is open by default, but you can collapse it by clicking the .
Scroll down and click the Run Fields section to open it.
Click Manually Define Fields.
One empty field is provided for you.
Enter the Name: "ProcessingLab" (without spaces).
Click Save.
Click Assay Tutorial to return to the main folder page.
Collect Metadata During Import
Adding a Run field has the effect of modifying the import wizard, providing an opportunity to collect the name of the processing lab for the file.
To see this in action, import new data into the Assay Design:
Upload them to the server using drag-and-drop into the Files panel (like you did in Step 1 of this tutorial).
Select the two new files using the checkboxes and click Import Data (like Step 3).
In the popup, select "Use Cell Culture" if it is not selected by default and click Import.
Click through the import wizard as previously, except this time enter values in the new Processing Lab field, such as "Jones Lab" or "Smith Lab".
Now you have a differentiator to compare results in your analyses and visualizations.
Congratulations
You have completed the Assay Tutorial and learned to import and analyze experiment results using LabKey assay tools. Next, explore some of the following features:
Accurate and consistent user entry is important to assay data, especially when it includes manual input of key metadata. For example, if an operator failed to enter a needed instrument setting, and later someone else wants to recreate an interesting result, it can be impossible to determine how that result was actually obtained. If an instrument's brand name is entered where a serial number is expected, results from different machines can be erroneously grouped as if they came from a single machine. If one machine is found to be faulty, you may be forced to throw out all data if you haven't accurately tracked where each run was done.This topic demonstrates a few of the options available for data validation during upload:
Required fields: prevent operators from skipping critical entries
If you wish to follow along exactly, execute the steps in the above tutorial before proceeding.
Set Up Validation
Here we add some validation to our GenericAssay design by modifying it. Remember that the assay design is like a map describing how to import and store data. When we change the map, any run data imported using the old design may no longer pass validation.Since the assay design was created at the project level, and could be in use by other subfolders, let's make a copy in the current folder to modify.
Open the design for editing:
Navigate to the Assay Tutorial folder.
In the Assay List web part, click Cell Culture.
Select Manage Assay Design > Copy assay design.
Click Copy to Current Folder (or click the current folder name on the list).
You will see the assay designer, with the same content as your original assay design.
Enter the Name: "Cell Culture Modified".
Required Fields
By default, any new field you add to an assay design is optional. If you wish, you can make one or more fields required, so that if an operator skips an entry, the upload fails.
Click the Run Fields section.
For the ProcessingLab field, check the Required checkbox.
Click Save.
Regular Expressions
Using a regular expression (RegEx) to check entered text is a flexible form of validation. You could compare text to an expected pattern, or in this example, we can check that special characters like angle brackets are not included in an email address (as could happen in a cut and paste of an email address from a contact list).
Click Cell Culture Modified in the Assay List.
Reopen Manage Assay Design > Edit assay design.
Click the Batch Fields section.
Click Add Field and enter the name "OperatorEmail" (no spaces).
Expand the field details.
Under Conditional Formatting and Validation Options, click Add Regex.
Enter the following parameters:
Regular Expression: .*[<>].*
Note that regex patterns are matched on the entire string submitted, so including ".*" delimiters and enclosing the pattern with [ ] (square brackets) will find the regex pattern within any longer string.
Description: Ensure no angle brackets.
Error Message: An email address cannot contain the "<" or ">" characters.
Check the box for Fail validation when pattern matches field value. Otherwise, you would be requiring that emails contained the offending characters.
By checking that a given numeric value falls within a given range, you can catch some bad runs at the very beginning of the import process.
Click Results Fields to open the section.
Find and expand the CultureDay field.
Under Conditional Formatting and Validation Options, click Add Range.
Enter the following parameters:
First Condition: Select Is Greater Than: 0
Second Condition: Select Is Less Than or Equal To: 10
Error Message: Valid Culture Day values are between 1 and 10.
Name: DayValidRange
Click Apply.
Scroll down and click Save when finished editing the assay design.
Observe Validation in Action
To see how data validation would screen for these issues, we'll intentionally upload some "bad" data which will fail the validation steps we just added.
Click Assay Tutorial to return to the main folder page.
Select Use Cell Culture Modified and click Import.
Paste in "John Doe <john.doe@email.com>" as the OperatorEmail. Leave other entries at their defaults.
Click Next.
Observe the next red error message: "Value 'John Doe <john.doe@email.com>' for field 'OperatorEmail' is invalid. An email address cannot contain the "<" or ">" characters.
Correct the email address entry to read only "john.doe@email.com".
Click Next again and you will proceed, no longer seeing the error.
Enter an Assay ID for the run, such as "BadRun".
Don't provide a Processing Lab value this time.
Click Save and Finish.
The sequence in which validators are run does not necessarily match their order in the design.
Observe the red error text: "ProcessingLab is required and must be of type String."
Enter a value and click Save and Finish again.
Observe error message: "Value '-2' for field 'CultureDay' is invalid. Valid Culture Day values are between 1 and 10." The invalid CultureDay value is included in the spreadsheet being imported, so the only way to clear this particular error would be to edit/save/reimport the spreadsheet.
There is no actual need to import bad data now that we have seen how it works, so cancel the import or simply click the Assay Tutorial link to return to the home page.
An administrator can set up a folder with the necessary web parts for assay/instrument data. You can either use the Assay folder type directly, or add the web parts to another folder type with tools you will use, such as Study.
Assay folder: Creating an Assay folder allows you to set up a staging area for assays, separate from other types of data. To integrate with Study data, you would create a different folder for the study.
Study folder: Creating a Study folder and adding assay web parts places all assay and study data in one place. If you do not care about separating assay data, you can choose this option.
Assay List Web Part
Users can access the Assay List by selecting > Manage Assays. You can make it more convenient by adding the Assay List web part to your folder.The Assay List is the starting place for defining or using assays available here. It lists assays defined in any of the following places:
Current folder
Current project
The Shared project, where assays are placed to be shared site wide. Learn more here.
Other Assay Web Parts
Additional assay web parts can be added to a folder to display information for specific assays:
Assay Batches: Displays a list of batches for a specific assay.
Assay Runs: Displays a list of runs for a specific assay.
Assay Results: Displays a list of results for a specific assay.
Files: While not exclusive to assays, providing a file browser can be useful. Learn more in this topic: Using the Files Repository.
Integrate with Study - Directly or by Linking
You can link quality-controlled assay results into a study when these results are ready for broader sharing and integration with other data types. The target study can exist in the same folder as your assay list or in a separate one. Assay results will be linked into a study and appear as datasets.If you plan to publish your assay data to a study, create or customize a study-type folder. You can include a specific study to which you want to link approved data in your assay design.If you want to avoid creating a separate study folder you may also enable study-features in an existing assay-type folder:
Assay designs are LabKey Server's way of capturing and importing instrument data from a wide variety of file output formats. The data structures of an assay design tell the server how to interpret the instrument output, so that the data can be uploaded 'into' the design and stored with metadata for further use.
When you create an assay design, you start from an assay type (usually "Standard"), give it a unique name, and then customize the defaults to fit your specific requirements.
Click New Assay Design in the Assay List web part (or via > Manage Assays).
By default, you will be on the Standard Assay tab.
If you are using a Premium Edition and want to create an instrument-specific design, click the Specialty Assays tab and select the desired format. For more details, see below
Select the Assay Location: This determines where the assay design is available.
You can scope your design to the current folder, the current project, or place it in the Shared project to make it available site wide. Note that if you are in a subfolder of a Biologics or Sample Manager project, assays will always be created at the project level.
Click Choose Standard Assay.
Define Assay Properties
Assay properties are set once for all uses of the design. For example, whether runs/results are editable and whether to enable setting of QC states.Review details about these properties in this topic: Assay Design Properties
Enter Assay Properties:
Enter a unique Name: the name is always required and while it may be changed after the design is created, it must remain unique.
The name must start with a letter or number character, and cannot contain special characters or some reserved substrings listed here.
Note that users with the Platform Developer role will see additional assay properties for working with transform scripts, not shown above.
Assay Fields
Each assay design consists of sets of fields, each capturing the contents of some individual column of uploaded assay data or metadata about it. Results/data fields can each have properties such as:
The data type
Whether a value is required
How to validate values and/or how to interpret special values
How to display the resulting data and use it in reporting
Some fields will be populated from the uploaded data itself, other values can be provided by the operator at import time.Fields are grouped into (at least) three categories:
Batch: A setting applies to an entire batch of runs (or files) uploaded at once. Example: the email of the person uploading the data.
Run: A setting applies to all the data in a single run, but may vary between them. Example: An instrument setting at the time of the run.
Results/Data: A setting applies only to a single row. Example: the ID of the participant each row of data is about.
Additional Categories: Some assay designs will include sections for plate data, analyte information, or other details.
Drag and drop to rearrange fields using the handles on the left.
If you need to remove a field, click the .
Open the settings and properties panel for fields using the (expansion) icon.
Learn more about editing properties of fields in this topic: Field Editor
Click Save when finished. Your new assay is now listed in the Assay List web part.
Once defined, you can import as many data files as you wish using this design.To edit, copy, delete or export an assay design, please see: Manage an Assay Design.
Assay Types
Assay designs are based on assay types, which are "templates" corresponding to a specific instrument or assay process.
Standard Assay (Recommended)
The most flexible type of assay is the "Standard Assay" (previously referred to as a "general purpose assay type" or GPAT). This type includes a minimal set of built in properties and supports a variety of extensions including:
Assay QC states and reporting
Plate support
The base properties in a Standard Assay are outlined in this topic:
Specialty Assays (Premium Feature or Developer-Defined)
Instrument-specific assay types pre-define many fields and properties appropriate for the particular instrument, which can be further refined as needed. If a developer has defined a custom module-based assay, it will also be listed on this tab. This does not require a premium edition, but does involve file-based module development. Learn more here.Properties for several of the built-in specialty types available with Premium Editions are outlined in these topics:
When creating a specialty assay, you will click the Specialty Assays tab and select the desired type from the Use Instrument Specific Data Format dropdown. You will set the location as for a standard assay and then click Choose [TYPE] Assay to define properties as described [above.
Other Ways to Create Assay Designs
Use an Existing Assay Design as a Template
You can also create a new assay design by copying from an existing design that is closer to your needs than any of the built in types. See Manage an Assay Design.
Infer an Assay Design from Spreadsheet Data
Another way to create an assay design is to infer it from a sample spreadsheet, then edit to change the fields and properties as needed.
Upload the sample spreadsheet to the file browser in the desired folder.
Select it and click Import Data.
In the popup, select the option you want, usually Create New Standard Assay Design.
Provide a name and adjust the information inferred from the file if needed.
In particular, if your field names include any special characters (including spaces) you should use the assay designer to adjust the inferral to give the field a more 'basic' name move the original name to the Label and Import Aliases field properties. For example, if your data includes a field named "CD4+ (cells/mm3)", you would put that string in both Label and Import Aliases but name the field "CD4" for best results.
Click Save and you will be taken to the first step of importing the data. You may Cancel at this point to save the new design without actually importing the data.
Note that LabKey does have a few reserved names in the assay framework, including "container, created, createdBy, modified, and modifiedBy", so if your spreadsheet contains these columns you may encounter mapping errors if you try to create a new assay from it. You can work around this issue during assay design creation by editing the names of these columns and using the Advanced > Import Aliases option to map the original name to the new name you added.
An authorized user designs an assay by creating a named instance of either the flexible "Standard" type or of a built-in "Specialty" assay type. In an assay design, you describe the structure of the data you will be importing and can customize it adding and modifying fields and properties as needed. This topic covers properties and fields available for Standard assay designs.
Assay properties describe attributes of an assay design that are common to all runs and batches imported using it. Every assay design must have a unique Name; other properties are optional and depend on the needs of the group using it. When you define an assay, the first panel includes properties in the following categories:
Basic Properties
Name(Required): Text. Each assay design must have a unique name.
The name must be unique, must start with a letter or number character, and cannot contain special characters or some reserved substrings listed here.
Description: Optional text with a more verbose description.
QC States: (Premium Feature/Available in Standard assays) Check to enable defining and using assay QC states for this assay design.
Editing Settings
By default, assay data cannot be edited after import into LabKey Server. It's often the case that the instrument generates the final, authoritative form of the data, which should not be changed after the fact. In cases where you want to allow authorized users to change assay run information or result data after import to the server, you can use these settings:
Editable Runs: If enabled, users with sufficient permissions can edit values at the run level after the initial import is complete. These changes will be audited.
Editable Results: If enabled, users with sufficient permissions can edit and delete at the individual results row level after the initial import is complete.
Note that new result rows cannot be added to existing runs.
Can Batch fields be edited? Yes, but only if you re-import a run. The import wizard allows you to update any batch fields.
Import Settings
Import in Background: If enabled, assay imports will be processed as jobs in the data pipeline. If there are any errors during import, they can be viewed from the log file for that job.
Transform Script Options (For Developers)
Users with the "Platform Developer" role (and Site Administrators) will have additional options in the Import Settings section for using transform scripts with their assay designs. Transform scripts run before the assay data is imported and can reshape the data file to match the expected import format. Learn more in this topic: Transform Scripts.
Transform Scripts:
Click Add Script to either directly upload a script file (via select or drag and drop) to the "@scripts" subdirectory of the file root, or you can also enter the full path to the transform script file located elsewhere.
For each script, you can select whether it is run on import, on edit, or both.
Multiple scripts can be attached to an assay design. They will run in the order specified.
Disassociate a script from an assay using > Remove Path; this does not delete the file.
Use Manage Script Files to access the "@scripts" location where you can delete or update script files.
Save Script Data for Debugging: Typically transform and validation data files are deleted on script completion. For debug purposes, it can be helpful to be able to view the files generated by the server that are passed to the script. If this checkbox is checked, files will be saved to a subfolder named: "TransformAndValidationFiles", located in under the file root of the folder.
Link to Study Settings
Auto-link Data to Study: If a target study is selected, then when new runs are imported, data rows are automatically linked to the specified target study (if they include subject and visit/date information). Learn more here: Link Assay Data into a Study
Linked Dataset Category: Specify the desired category for the Assay Dataset that will be created (or appended to) in the target study when rows are linked. Learn more here: Manage Categories.
Batch Fields
The user is prompted for batch properties once for each set of runs during import. The batch is a convenience to let users set properties once and import many runs using the same suite of properties. Typically, batch properties are properties that rarely change. Default properties:
Participant Visit Resolver This field records the method used to associate the assay with participant/visit pairs. The user chooses a method of association during the assay import process. See also Participant/Visit Resolver Field.
TargetStudy. If this assay data is linked to a study, it will go to this study. This is the only pre-defined Batch property field for Standard Assays. It is optional, but including it simplifies the link-to-study process. Alternatively, you can create a property with the same name and type at the run level so you can then publish each run to a different study. Note that "TargetStudy" is a special property which is handled differently than other properties.
Run Fields
Run properties are set once for all data records imported as part of a given run. An example run property might be the ID of the instrument or the operator performing the upload.No default run properties are defined for Standard Assays.
Results Fields
Results fields (also can be called data properties) apply to individual rows within the imported run.The pre-defined Results Fields fields for Standard Assays are:
SpecimenID
ParticipantID
VisitID
Date
These properties are used to associate assay data with other data from the same source material. For more, see Participant/Visit Resolver Field.In addition, there are Created/CreatedBy/Modified/ModifiedBy fields built into most LabKey data structures, including Assay Results. Exposing these fields in a custom grid view can help track changes, particularly when Editable Results are enabled for the Assay Design.
Files and Attachments
Assay designs support associating a given row of data with a file using either a run field or results field of one of these types:
File: A field that creates a link to a file. The file will be stored in the file root on the server, and will be associated with an assay result.
Attachment: A field that will associate an image file with a row of data in a list.
These files might contain images or rectangular data. For example, to index microscopy files, you might create an assay design with metadata and descriptive fields (such as content, timing, staining) and then include an attachment file with the image.
Some assays use plate-based technologies where spots or beads of a sample are arrayed across a fixed size plate and read by an instrument. Wells in the plate, typically referred to by column and row, contain different dilutions or combinations of elements and data is recorded by well. Creating an assay design for a plate-based technology includes the creation of one or more plate templates to the assay design process outlined in Design a New Assay.When you create an assay design, you choose the assay type up front and then can customize a specific design to match your experiment data.
Premium Features AvailableSubscribers to premium editions of LabKey Server can use built in Specialty Assays which include corresponding instrument-specific default plate layouts. Learn more in these topics:
Plate Templates describe the layout of wells on the plate read by a given instrument. Each well can be associated with experimental groups describing what is being tested where and how the data read should be interpreted. Different instruments and experiments use different configurations of these groups.Plate templates are used to describe the specifics of each layout for interpretation within an assay design. Using a custom plate template, you can create the precise match you need to describe your own exact configuration of wells and roles.
Select > Manage Assays (or click Manage Assays in the Assay List web part) to see the list of currently defined assays.
Click Configure Plate Templates to open the Plate Templates page.
All plate templates currently defined (if any) are listed, along with the type and how many usages of each. For each, you can:
Edit: Open this template in the plate template editor.
Note that you cannot edit a template after it has been used to import data.
Edit a copy: This option opens a copy of the template for editing, leaving the original unchanged.
Copy to another folder: Click, then select the target folder.
Delete: Only available if more than one template is defined. Once any templates are defined, you cannot delete all of them.
From the Plate Templates page you can also create a new template from any one of the available built-in types.
Select the desired type from the dropdown below the currently defined templates.
Click Create.
Premium Features AvailableSubscribers to premium editions of LabKey Server can use a variety of built-in Specialty Assays, some of which have corresponding custom plate templates to start from. Learn more here:
The Plate Template editor lets you lay out the design of your experiment by associating plate wells with experimental groups. An individual cell might be associated with multiple groups of different types, such as different viruses applied to both controls and specimens. This walkthrough uses the 96-well Standard template as a representative example with some predefined groups.Each tab in the plate template editor can define a different set of groups, properties, and layouts to meet your experiment needs.When you wish to save your changes, you can click Save and continue to edit. When you have finished editing, click Save & Close to exit the template editor.
Create a Plate Template
Select > Manage Assays.
Click Configure Plate Templates.
Select "New 96 Well (8x12) Standard blank template" from the dropdown.
Click Create.
Enter a unique Template Name. This is required even if you make no changes to the default layout.
Click Save and complete the template as described below.
Use Well Groups
Plate templates can have multiple tabs for different "layers" on the plate, or aspects of your layout. Shown in the Standard example here are Control and Sample. In premium editions, some Specialty plate templates have additional layers.Below the grid, you will see any well groups defined. Color coding distinguishes them on the layout. Colors are assigned arbitrarily and will be unique for every group in a template but cannot be individually set or changed.
On the Control tab of the Standard blank template, you have predefined well groups for "Positive" and "Negative" controls.
You could add additional control well groups following the steps below.
Select one of these well-groups and click (or "paint" by dragging) to identify the wells where those controls will be on your plate.
Shown here we have column 1 as the Negative control and column 2 as the Positive control.
Create and Edit Well Groups
On the Sample tab, there are no predefined well groups for the Standard blank template.
Add additional groups by entering a group name in the New box and clicking Create. For example, you might want to have 5 different samples tested on each plate. You can add each group individually:Or add multiple groups at once by clicking Create Multiple.... You can create as many groups as you like sharing a common name root. In this example, 5 "Sample" groups are created:Select any well group by clicking the name below the grid. Once a group is selected you can:
The Up, Down, Left, and Right buttons can be used to shift the entire layout if desired.
Define and Use Well Group Properties
In the section on the right, click the Well Group Properties tab to see any currently defined properties for this group. You can enter a value, use the to delete, or click to Add a new property.For example, some assays assume that samples get more dilute as you move up or left across the plate, others reverse this direction. Adding a well group property named 'ReverseDilutionDirection' will support controlling this default behavior. Enter the value 'true' to reverse this default behavior for a given well group.
View Warnings
If any Warnings exist, for example, if you identify a single well as belonging to both a sample and control group, the tab label will be red with an indication of how many warnings exist. Click the tab to see the warnings.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Several types of assays use plate-based technologies where spots or beads of sample are arrayed across a fixed size plate and read by an instrument.When you create an assay design, you choose the assay type up front and then can customize a specific design to match your experiment data. Premium Editions of LabKey Server include several built-in Specialty Assay types for plate-based instruments, which include corresponding built in default plate layouts, as covered in this topic.Creating an assay design for a plate-based technology adds the creation of one or more plate templates to the general assay procedure outlined in Design a New Assay.
For each assay, you will first define a plate template. The general instructions for working with Standard plate templates are described in this topic.
In a Premium Edition where Specialty Assays are defined, the list of options is much longer, offering instrument-specific templates to get you started.Some types have additional layers or sets of well groupings. Shown below, a neutralizing antibody assay has tabs (layers) for Control, Virus, Specimen, Replicate, and Other well groupings, with defaults that make sense for this type of instrument.On each tab, you can add additional groups and "paint" the wells where those groups are represented.Learn more in the documentation for your specialty assay type.
Design Specialty Plate-Based Assays
When you create an assay design from one of the built-in plate-based types, you associate it with a single template. If you want to be able to use multiple templates, you must design multiple versions of the assay design.
From the Assay List, click New Assay Design.
Click the Specialty Assay tab.
Select the desired assay type (ELISA, ELIspot, or NAb) from the Use Instrument Specific Data Format dropdown.
Specify the Assay Location.
Click Choose [Instrument Type] Assay to open the assay designer.
Name the assay design (required).
The name must be unique, must start with a letter or number character, and cannot contain special characters or some reserved substrings listed here.
The Plate Template menu will include all templates available in this location. Select the template that this assay design should use.
If you have not yet defined your plate, you can click Configure Templates here and define a new one. Note that this action will exit the assay designer, discarding any information entered so far.
Make other assay design changes as required.
Click Save.
Using Specialty Plate-Based Assays
When data is uploaded for a plate-based assay of one of the Specialty types, the plate template attached to the assay design is used. The user will not see the template name or details, nor have any opportunity to change it. If a change to the plate design is needed in the future, an administrator or assay designer must edit the assay design to change which template is used.For a walkthrough of using a plate template with a built in assay design type, try one of these tutorials:
Assay and instrument data can be integrated with other datasets in a study provided there is a way to resolve it to the participants and visit/timepoint information used in those datasets. When you import most kinds of assay data, you can select a Participant/Visit Resolver, to define how the assay data will be mapped and integrated.
Assay and instrument data often includes a Sample ID field, but not Participant ID or Visit fields. In many cases, this is "by design", since it allows laboratories to work with research data in a de-identified form: the laboratory only knows the Sample ID, but does not know the Participant IDs or Visit IDs directly.This topic outlines some general principles and options available for data identifiers; specific options available vary based on the type of assay.
Participant/Visit Resolver
When importing runs of instrument data in the user interface, one of the Batch Properties lets the user select from a set of options which may include some or all of the following:
Sample information in the data file (may be blank).
Participant id and visit id.
Participant id and date.
Participant id, visit id, and date.
Specimen/sample id.
Sample indices, which map to values in a different data source.
Note that not all assay types include all options.
Map Sample Indices
The Sample indices, which map to values in a different data source option allows you to use an existing indexed list of participant, visit/date, and sample information, such as a thaw list. At upload time, the user will enter a single index number for each sample; the target data source will contain the required mapping values. The sample indices list must have your own specimen identifier (index) as its primary key, and include the 'SpecimenID', 'ParticipantID', 'Date', and 'VisitID' columns.
You can specify a mapping either by pasting a TSV file or by selecting a specific folder, schema, and list. Either method can be used during each upload or specified as a default. To paste a TSV file containing the mapping, you can first click Download Template to obtain a template. After populating it with your data, cut and paste the entire spreadsheet (including column headers) into the box provided:
To specify an existing list, use the selection dialog pulldowns to choose the folder, schema, and list (or query) containing your mapping:
Use Default Values
The operator may specify the mapping each time data is uploaded, but in some cases you may want to set automatic defaults. For example, you might always want to use a specific source list for the participant/visit identifier, such as a thaw list populated at the time samples are removed from the freezer for testing. The operator could specify the list at the time of each batch upload, but by including the default list as part of your assay design you can simplify upload and improve consistency.
This topic describes options for managing assay designs. Assay designs can be defined in a local container, or shared from a project or sitewide from the Shared project. Be aware that editing a design shared in other locations will change it for all users. Another alternative is to edit a copy of the design.
Open the list of currently defined assays by selecting > Manage Assays. Click on the name of any assay to open the runs page. The Manage Assay Design menu provides the following options:
Edit assay design: Add, delete, or change properties or structure.
The Name is always required.
For Standard assay designs the Name can be edited after creation, but must remain unique. For Specialty assay designs, editing the Name after creation is not allowed.
Note that all current users of the assay design, including those in subfolders, will be impacted by these changes.
Copy assay design: This option lets you create a new assay design based on the current one.
Changes to the copy do not affect the original design or runs that use it.
You can create the copy in the same location or choose another folder. Note that in a Biologics or Sample Manager project, assays can only be created at the project level.
Delete assay design: This option will delete the current design as well as all runs that used it. You will see a list of what would be deleted and have a chance to confirm or cancel this action.
Export assay design: The design will be exported to a XAR file you can import elsewhere. See Export/Import Assay Design.
(Premium feature) Create QC trend report: Create a QC trend report to track metrics relevant to your instrument. Learn more in this topic: Quality Control Trend Reports.
Set Default Values
Fields in an assay design can support defining default values using the field editor. Defaults can be of several types: last entered, editable default, and a fixed value.Values for editable and fixed value properties are then set using the Set Default Values option on the Manage Assay Design menu. The assay design may be then be inherited in subfolders, which may override these parent defaults if needed using the Set Default Values option in the other container. These folder defaults will, in turn, be inherited by sub-folders that do not specify their own defaults.
You can set defaults for:
Batch fields
Run fields
Properties specific to the assay type. For example, for an Luminex assay, additional items would include "analyte" and "Excel run file" properties.
Assay Data Auditing and Tracking Changes
Some assays, like the Standard assay type, allow you to make run and data rows editable individually. Editability at the run or result level is enabled in the assay design by an administrator. Any edits are audited, with values before and after the change being captured. See Assay/Experiment events in the audit log. Upon deleting assay data, the audit log records that a deletion has occurred, but does not record what data was deleted.Some assay types, including Standard and Luminex, allow you to upload a replacement copy of a file/run. This process is called "re-import" of assay data. The server retains the previous copy and the new one, allowing you to review any differences.See the Assay Feature Matrix for details about which assay types support editable runs/results and re-import.
This topic explains how to export and import assay designs using a pre-prepared assay design file, or XAR/xar.xml file. Assay design import/export is not available for plate-based assays that use templates (such as NAb and ELISpot), but it is available for Standard assays and Luminex assays. Import/export does not support transform scripts, but does support validation properties (regular expressions and range checks on fields).To learn about exporting and importing assay data, also using a XAR file format, review this topic: Export Assay Data.
Export Assay Design Archive
From the batch, run, or results grid of any supported assay, select Manage Assay Design > Export assay design to export the design as a XAR file. The .xar file will be downloaded directly.
Import Assay Design Archive (.xar)
Click New Assay Design in the Assay List (or via > Manage Assays).
Click the Import Assay Design tab.
You have two options here:
Drag the .XAR (or .XAR.XML) file into the target area.
Click Import.
OR
If your XAR file has already been uploaded to a location visible to this folder's data pipeline, you can instead click Use data pipeline.
You will see the file browser.
Locate and select the XAR file and click Import Data.
In the popup dialog select Import Experiment and click Import.
Once the import has completed, refresh your Assay List web part to see the new assay on the list of available designs. If it does not appear immediately, it is still being uploaded, so wait a moment and refresh your browser window again.You can now import individual run data files to the assay design.
Example
An example XAR file is included in the LabKeyDemoFiles at LabKeyDemoFiles/Assays/Generic/GenericAssayShortcut.xar.Download: LabKeyDemoFiles.zip.
Export / Import a Folder: Assay designs that have been used to import at least one run will be included if you export a folder archive.
Assay QC States: Admin Guide
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Assay QC states can be defined for use in approval workflows. Administrators define the states and can enable using QC states on assays which support them, including customized general purpose assays (GPAT). Admins also assign the QC Analyst role to any users who will be setting these states.This topic covers the administrator tasks associated with Assay QC States. Learn about using the states in the topic: Assay QC States: User Guide.
Do not use Assay QC in a study folder. The use of QC states in studies for dataset QC may conflict with use for assays. To combine Assay QC and study functionality, define assays and studies in separate folders.
Assign the QC Analyst Role
In order to assign Assay QC states, users need the QC Analyst role, which is assigned in the project or folder where it applies. Note that the QC Analyst role does not include Reader (or other) permissions automatically. Users must also have at least the Reader role in the folder.
Select > Folder > Permissions.
Add the users or groups to the QC Analyst role.
Confirm that the users also have at least the Reader role in addition to the QC Analyst role.
Click Save and Finish.
Enable Assay QC
You must enable the use of QC States on at least one assay in a container in order to define or use states. You can enable this option during assay creation, or add it to an existing assay design created using the general purpose type.
Open the assay designer:
To start creating a new assay design, select > Manage Assays > New Assay Design, choose the "General" type and folder location, and click Next.
In the General Assay Designer, Assay Properties section, check the box for QC States.
Click Save when finished.
Define Assay QC States
The assay design will now have an additional QC State option on the grid menu. An Admin can define the desired QC states for the container by selecting QC State > Manage states. Note that QC Analysts will not be able to see this option on the menu. Also note that the QC States listed here include all of the dataset and assay QC states defined in the same folder.
Viewing the assay runs grid, select QC State > Manage states.
Click the green Add State icon to define the states your workflow requires. For example, you might define "Not Yet Reviewed", "Reviewed - Passed", and "Reviewed - Failed" for a simple workflow.
For each state, enter:
State Name: This name will be presented on the menu for users setting states.
State Description: For your reference. This description is not displayed for the user.
Public data: Checking this box means that the data in this state will be visible to data readers (i.e., possessors of the Reader role). For example, to hide runs that failed QC, you would uncheck this box for the 'Reviewed - Rejected' state.
The In Use column indicates whether the state is in use for any assays in the folder.
If you make a mistake, you can delete any row by clicking the red on the right.
After defining the states you will need, click Save.
QC Analysts can see all the assay data in a folder, even that which has been assigned to QC states that are not "public", i.e. are not visible to non-analyst readers in the container.
Manage QC States
Once states are defined, when you reopen the QC State > Manage states menu you will be able to set a default and make other changes.
Set a Default
There is a predefined "[none]" state that is the default default, but in the Default states for assay data section you can select the state that newly imported data will have. Note that you will not be able to set a default QC state until after you have saved the set of states.
From the assay runs grid, select QC State > Manage states.
Under Default states for assay data, select the desired state from the set defined.
Note that the dropdown list will only included states previously saved. If you add a new state, it will not appear on this list until you have saved and reopened this page.
Click Save.
In Use Status
The In Use status column is set true when any runs in the container are set to that specific state. The "[none]" built in state cannot be deleted, so is always marked as "in use" even if the folder does not contain any runs, or contains only runs set to other states.
Delete Unused States
Cleaning up an outdated set of no longer in use states, or changing the set available can be done by deleting any states not in use. You can delete states individually or all at once.
From the assay runs grid, select QC State > Manage states.
Click the red on the right to delete any row.
If you are deleting a row that has been saved, you will be asked to confirm the action.
If you are deleting rows before you have clicked Save, no confirmation will be needed.
Click Delete Unused QC States to delete all states not currently in use. You will be asked to confirm this action.
Do not delete Assay QC states in a study folder. States that appear unused for Assay QC may be in use for dataset QC.To combine Assay QC and study functionality, define assays and studies in separate folders.
Require Comments for QC State Changes
To require that users supply a comment whenever the QC state of a run is changed, select Yes under QC State Comments. This setting applies to all assays using QC states in the current container.This will make the comment field required when a user changes changes the QC state.To see the history of state changes and comments for a given run, click the value in the QC Flags column. The QC History of all changes is shown below the change panel.
LabKey's assay framework helps you to share experimental data and metadata with collaborators. It can be a powerful tool for aggregating data across multiple labs and for making decisions on a course of research based on what others are finding. But how can you record data in a way that makes it easily comparable across labs? When different groups use slightly different words for the same thing, how can you ensure that data are entered consistently? How can you guard against the inevitable typo, or entry of the wrong information into the wrong field?This page introduces a few of the ways LabKey Server can help your team improve consistency and reduce user error during initial data entry:
When users upload assay data, they often need to enter information about the data and might use different names for the same thing. For instance, one user might enter "ABI-Qstar" and another simply "Qstar" for the same machine. By defining a lookup for an assay field, you can eliminate this confusion by only allowing a pre-set vocabulary of options for that field.In this scenario, we want users to choose from a dropdown list of instruments, rather than name them instrument themselves when they upload a run. We modify the assay design to constrain the instrument field to only values available on a given list. The example "Cell Culture" design used here comes from the design a general purpose assay tutorial, so if you have completed that tutorial you may follow these steps yourself.
Drag and drop the "Instruments.xls" file into the target area.
Once the fields are inferred, select "InstrumentID" from the Key Field Name dropdown in the blue panel.
Confirm that the Import Data checkbox is checked (it should show three rows of your data).
Click Save.
You can click the name Lab Instruments to see the list of options that will be presented by the lookup.
Add Lookup to Assay Design
Change the assay design so that the instruments field no longer asks for open user entry, but is a lookup dropdown instead:
Select > Manage Assays.
Click Cell Culture.
Select Manage Assay Design > Edit assay design.
Click the Batch Fields section to open it.
Click Add Field and enter the Name: "Instrument".
Use the Data Type dropdown menu to select Lookup.
The panel will open, so you can enter the Lookup Definition Options:
Target Folder: [current folder]
Target Schema: lists
Target Table: Instruments (String)
Check the box for Ensure Value Exists in Lookup Target to require that the selected value is present in the lookup's target table or query.
Scroll down and click Save.
Usage Demonstration
If you have now made these changes in your tutorial project, you can see how it will work by starting to import an additional run:
On the Cell Culture Runs page:
Click Import Data.
On the Batch Properties page notice the new Instruments field is a dropdown list.
Note: if the lookup target contains 10,000 rows or more, the lookup will not be rendered as a dropdown, but as a text entry panel instead.The system will attempt to resolve the text input value against the lookup target's primary key column or display value. For matching values, this will result in the same import behavior as if it were selected via dropdown menu.
Set Default Values
When the user must enter the same fixed values repeatedly, or you want to allow prior entries to become new defaults for given fields, you can do so using built in default values for fields. Default values may be scoped to a specific folder or subfolder, which can be useful in a situation where an assay design is defined at the project level, the overall design can be shared among many subfolders, each of which may have different default value requirements.
Field level validation can programmatically ensure that specific fields are required, entries follow given regular expressions, or are within valid ranges.
It can be useful to transform data columns during the process of importing data to LabKey Server. For example, you can add a column that is calculated from other columns in the dataset. For this simple example, we use Perl to add a column that pulls the day out of a date value in the dataset. This example builds on the design used in the Assay Tutorial.
To use a transform script in an assay design, edit the design and click Add Script next to the Transform Scripts field. Note that you must have Platform Developer or Site Administrator to see or use this option.To use this example as written, start from the Assay Tutorial folder and make a copy of the CellCulture design you created during the Assay Tutorial.
On the main folder page, in the Files web part, select the file CellCulture_001.xls.
Click Import Data.
Select Use CellCulture_Transformed and click Import.
Click Next.
Click Save and Finish.
The transform script is run during data import and populates the "MonthDay" column that is not part of the original data file. This simple script uses perl string manipulation to pull the day portion out of the Date column and imports it as an integer. You can alter the script to do more advanced manipulations as you like.
Instrument data becomes even more useful when integrated with other data about the same participants in a LabKey Study. For example, assay data can track how a blood marker changes over time. After integration, you can compare trends in that marker for patients receiving different treatments and draw more useful conclusions.
Assay data records are mapped to the study's participants and visits. Within the study, a new dataset is created which includes the assay data using a lookup. This allows you to:
Easily integrate assay data with clinical data.
Create a broad range of visualizations, such as time charts.
Utilize qc and workflow tools.
Manual Link of Assay Data to Study
Assay data that has already been uploaded can be linked to a study manually. You can link individual result rows, or select all the results in a run by selecting by run. Note that when linking large assays, selecting by run will give you more options for managing larger bulk linking operations.
Navigate to the view of runs or results containing the data you want to link.
You can also specify a category for the linked dataset if desired. By default the new dataset will be in the "Uncategorized" category and can be manually reassigned later.
Click Next.
In the Sample/Specimen Match column, each row will show a (check) or icon indicating whether it can be resolved to a sample (or specimen) in that study.
If your assay data includes a field of type "Sample", linking the assay results to Samples, then this column will be named Sample Match.
Seeing the icon does not prevent linking assay data.
You will see the new assay dataset within the target study. It is named for the assay design and now connected to the other participants in the study.
Click a participant ID to see other information about that participant. (Then go back in your browser to return to this linked dataset.)
Notice the View Source Assay link above the grid. Click it to return to your original assay folder.
Click View Link to Study History to see how the linkage you just made was recorded.
You can also see which rows were linked in the grid view of the assay data. Click "linked" to navigate to the dataset in the target study.
Date Fields During Linking
When linking assay data to a date-based study, the system looks for a field named Date. If none is found, the system will look for another column of type "DateTime", i.e. a DrawDate column might be present. Provided only one such column is found, it will be used for linking. If there are two or more "DateTime" columns, you may need to manually reenter the desired Date values for successful linking to study.
If your Date column data includes a timestamp, you will only see and link the date portion by default (the time portion will become 00:00 in the linked dataset). To include the timestamp portion, click Display DateTime link above the listing. This will add the time to the Date field so it will be included in the linked version. Note that whether or not the timestamp is displayed is controlled by date display format settings.
Visit Fields During Linking
When linking assay data to a visit-based study, the system looks for a field named VisitId. If your assay uses another field name, you may have to provide visit identifiers manually.
Automatic Link-to-Study upon Import
When you always plan to link data to the same study, you can automate the above manual process by building it directly into the assay design. Each run uploaded will then be automatically linked to the target study. You can set this field during assay creation, or edit an existing design as described here.
In your assay folder, click the name of your assay design.
Select Manage Assay Design > Edit Assay Design. If you want to preserve the original version of the design that does not do the link-to-study, select "Copy Assay Design" instead and copy it to the current folder, giving it a new name.
From the dropdown for Auto-Link Data to Study, select the target study.
See below for information about the "(Data import folder)" option, if present.
If desired, set a Linked Dataset Category for the linked dataset. Learn more in this topic.
Scroll down and click Save.
The next data imported using this assay design will have all rows linked to the target study.
You can find "linked-to-study" datasets in a study on the Clinical and Assay Data tab. Unless otherwise assigned a category, datasets added from assays will be "Uncategorized."
Shared Assays: Auto Link with Flexible Study Target
If your assay design is defined at the project level or in the Shared project, it could potentially be used in many folders on your site.
You can set a single auto-link study destination as described above if all users of this assay design will consolidate linked data in a single study folder.
You may wish to configure it to always link data to a study, but not necessarily always to the same destination study. For this situation, choose the folder option "(Data import folder)". This will configure this assay design to auto link imported assay data to the study in the same folder where the data is imported.
You can then use this assay design throughout the scope where it is defined.
If it is run in a folder that has both assay and study tools available, imported assay data will automatically be linked to that study.
If it is used in a folder that is not also a study, the assay functionality will still succeed, but an error message about the inability to link to the study that doesn't exist will be logged.
Skip Verification Page (For Large Assay Files)
When your assay data has a large number of rows (thousands or millions of rows), the verification page can be impractical: it will a take a long time to render and to manually review each row. In these cases, we recommend skipping the verification page as follows:
Navigate to the Runs view of the assay data. (The Results or Batches view of the data will not work.)
Select the runs to be linked.
Click Link to Study.
Select the checkbox Auto link data from ## runs (## rows).
This checkbox is available only when you initiate link-to-study from the Runs view of the assay data. If you don't see this checkbox, cancel the wizard, and return to the Runs view of the assay data.
You will see both the number of runs you selected and the total number of result rows included.
Click Next.
The data will be linked to the study skipping the manual verification page. The linkage will run as a pipeline job in the background. Any assay data that is missing valid participant ids or timepoints will be ignored.
Recall Linked Rows
If you have edit permission on the linked dataset in the target study, you have the option to "recall" the linked row(s). Select one or more rows and click Recall. When rows are recalled, they are deleted from the target study dataset.The recall action is recorded on the "View Link to Study History" page in the assay folder.
Hover over the leftmost column in the grid for a (details) icon. Click it for more details about that row.
What Do Checks and Xs Mean?
The (check) or icons when linking are relevant to Sample or Specimen matches only. An X does not necessarily mean the link-to-study process should not proceed or does not make sense. For example, Xs will always be shown when linking into a study that does not contain specimens or refer to samples.When assay data is linked to a study, it must be resolved to participants and dates or visits in the target study. There are several ways to provide this information, and participants and visits that do not already exist will be created. See Participant/Visit Resolver Field. Matching incoming assay data to Samples or Specimens is done when possible, but integrated assay data is still of use without such a match.If the server can resolve a given ParticipantId/Visit pair in the assay data with a matching ParticipantId/Visit pair in the study, then it will incorporate the assay data into the existing visit structure of the study. If the server cannot resolve a given Participant/Visit pair in the assay data, then it will create a new participant and/or a new visit in the study to accommodate the new, incoming data from the assay. The assay data is linked into the study in either case, the difference is that in one case (when the assay data can be resolved) no new Visits or Participants are created in the study, in the other case (when the assay data cannot be resolved) new Visits and Participants will be created in the study.
Troubleshooting
Large Assay Link Operations
If you encounter errors during large link operations from the user interface, you may be attempting to link too many rows at once in one POST request. The upper limit is in the range of 9950 rows.To workaround this limitation you can try the following:
Split a manual link operation into several smaller chunks.
If you are linking a large number of assay result rows, try starting from the Runs list instead to link the runs containing those results. You will then have the option to skip the verification page.
If your data includes participant resolver information, use automatically link to study from within the data import wizard to bypass the manual import.
If you routinely have many rows to link to the same study, do not need to interact with the data between import and link, and your data always includes the necessary participant resolver information, you can also enable the automatically link to study option in your assay design directly.
After completing a link-to-study by any method, you'll be able to review the set of result rows that were successfully linked on the Link-to-Study History page.
Name Conflicts
In a study, you cannot have a dataset with the same name as an existing dataset or other table. This includes built in tables like "Cohort", "Location", and a table created for the "Subject Noun Singular" which is "Participant" by default. It is very unlikely that you would have an assay with the same name as one of these tables, but if you do, linking to study will fail with an error similar to:
Cannot invoke "org.labkey.api.data.TableInfo.getDefaultVisibleColumns()" because "tableInfo" is null
The exception to this name-conflict rule about is that if you have already linked rows from another assay with the same name, the new link action will successfully create a new dataset appending a 1 to that name.
Once either assay records or samples have been linked to a study, an administrator can view the logs of link-to-study events for these assays and sample types. The administrator can also recall, or 'undo', the linkages when necessary.
After users have linked assay or sample data to a study, an admin can view link-to-study history from either the source (assay or sample type) or from the destination (dataset grid).
From the Target Study Dataset
To access link-to-study history from a study dataset representing that linked assay or sample data, click View Source Assay or View Source Sample Type respectively above the grid, then proceed as below.
From the Assay
From a datagrid view for the assay, click View Link To Study History:The history is displayed in a grid showing who linked which runs to which study.
From the Sample Type
From the detail view for the Sample Type, click Link to Study History.As for an assay, the history is displayed in a grid showing who linked which samples to which study.
From the Site Admin Console
If you are a site administrator, you can also view all link-to-study events for all assays and samples within the site. You may notice that both "Assay/Protocol" and "Sample Type ID" columns are included in history grids, with only the relevant column populated for either type of linkage.
Select > Site > Admin Console.
Under Management, click Audit Log.
Select "Link to Study Events" from the dropdown.
Filtering this combined audit log by container can give you aggregated link histories for a study combining both assays and samples. You can also filter by assay or sample type across all studies and do other filtering and querying as with other data grids.
View Link-to-Study History Details
Once you have reached a Link-To-Study History page, hover over a row and click the (details) icon to see all the rows linked from the assay:You now see the "Link To Study History Details" page:
Recall Linked Data (Undo Linkage)
You can recall linked assay or sample data from a dataset, essentially undoing the linkage.
On the History Details page, select the rows that you would like to remove from the dataset and click Recall Rows.
You can also select rows in the study dataset and click Recall.
Click OK in the popup to confirm.
Rows recalled from the study dataset are removed from that target dataset, but are not deleted from the source assay or sample type itself. You can link these rows to the dataset again if needed.Recall events are audited and will appear in the Link-To-Study History.
This topic covers the tasks performed by a user (not an administrator) in an Assay folder configured for processing instrument data. You'll find a review of terms used here: Assay Terminology, and can review the topics in Assay Administrator Guide to learn how an administrator sets up this functionality.The best place to start learning about using LabKey assays is to begin with this tutorial which walks through the basic process for setting up and using a standard assay.
The import process for instrument data using an assay design involves some steps that are consistent for all types of assays. The process can vary a bit depending on the the type of assay and features enabled. This topic covers the common steps for importing assay data.
Before you import data into an assay design, you need a target assay design in your project. An assay design will correctly map the instrument data to how it will appear on the server.If you don't already have an assay design in your project, an administrator or assay designer can create one following one of these processes. Only Admins and users with the "Assay Designer" role can design assays.
Once you have an assay design you can begin the process of uploading and importing the data into the design. In the following examples, the assay design is named "GenericAssay".
Before uploading data files, consider how your files are named, in order to take advantage of LabKey Server's file grouping feature.When you import data files into an assay design, LabKey Server tries to group together files that have the same name (but different file extensions). For example, if you are importing an assay data file named MyAssayRun1.csv, LabKey will group it together with other files (such as JPEGs, CQ files, metadata files, etc.), provided they have the same base name "MyAssayRun1" as the data record file.Files grouped together in this way will be rendered together in the graphical flow chart (see below) and they can be exported together as a zip file.
Import Data into an Assay
In the Files web part, navigate to and select the files you want to import.
Click Import Data.
In the popup dialog, select the target assay design and click Import.
Enter Batch Properties
You are now on the page titled Data Import: Batch Properties. A batch is a group of runs, and batch properties will be used as metadata for all runs imported as part of this group. You can define which batch properties appear on this page when you design a new assay.
Enter any assay-specific properties for the batch.
Click Next.
Enter Run-Specific Properties and Import Data
The next set of properties are collected per-run, and while there are some commonalities, they can vary widely by type of assay, so please see the pages appropriate for your instrument. Review run-specific properties and documentation for importing data appropriate for your assay type:
When you import an assay run, two new assay events are created:
Assay Data Loaded: This event records the new run you imported.
Added to Run Group: The new run is added to the batch with other runs in the same import.
Assay Data and Original Files
In the case of a successful import, the original Excel file will be attached to the run so you can refer back to it. The easiest way to get to it is usually to click on the flow chart icon for the run in the grid view. You'll see all of the related files on that page.Once you have imported a file you cannot import it, or another file of the same name, using the same method. If you need to repeat an import, either because the data in the file has been updated or to apply an updated import process, use the process covered in Re-import Assay Runs.For failed import attempts, the server will leave a copy of the file at yourFolder/assaydata/uploadTemp. The subdirectories under uploadTemp have GUIDs for names -- identify your file based on the created/modified date on the directory itself.
A single assay run may span multiple files if there are many unknowns on a single plate, or if the standards and unknowns are stored in separate files but are still considered part of the same run.Some Assay Types support multi-file runs. The options for importing multiple files as a single run are covered in this topic.
Begin the import from the assay design, not from the pipeline file browser.
Select > Manage Assays.
Click the name of your assay design to open it.
Click Import Data above the runs grid.
Specify the necessary batch and run properties as prompted. If you don't supply a name for the run, the name of the first file selected to import will be used.
When you get to the Run Data item, use the Upload one or more data files option.
Click Choose File and you can multi-select more than one file in the file picker. Click Open when all files are selected.
You will see the selected filenames shown, along with the number chosen.
If needed, you can use the (plus) icon to add additional files. Note that if you attempt to add a file that has already been added, the server will raise an alert and not add the duplicate file to the list.
Click Save and Finish to complete the import.
Drag and Drop to Import a Multi-File Run
Another option is to drag and drop the necessary files onto the Choose File button.Note that while drag and drop of directories is supported in some file pickers, you cannot import a directory of files to the assay importer.
Import Multiple Multi-File Runs
Follow either import option above for the first run, then instead of finishing, click Save and Import Another Run to add as many more multi-file runs as needed to this batch.For the next run, you can use the same process to multi select as many files as comprise the run.
There is no requirement that all runs in a batch contain the same number of files per run. You can combine single-file runs with multi-file runs as once the import of data is complete, the runs will be treated equivalently on the server.
Import Multiple Single-File Runs
If you initate assay import from the file browser and select multiple files, as shown below, they will each be imported as separate runs as part of a single batch. This option is illustrated here to differentiate from the above options for single runs composed of multiple files.You can also accomplish the import of a batch of single-file runs by selecting or dragging a single file within the assay import dialog as described above.
To reach the grid of assay runs, select > Manage Assays. Click the name of your assay. You can also reach this page from other views within the assay data by clicking View Runs.You'll see the runs grid listing all runs imported so far for the given assay design. Each line lists a run and represents a group of data records imported together.To see and explore the data records for a particular run, click the run's name in the Assay Id column. If you didn't explicitly give the run a name, the file name will be used.You'll see the results from that particular run. Learn about the assay results view in this tutorial step: Step 4: Visualize Assay Results.Return to the runs view by clicking View Runs above the results grid.
Flag for Review
Flagging an assay run gives other users a quick visual indicator of something to check about that run. To flag a run, click the (Flag) icon.You will be prompted to enter a comment (or accept the "Flagged for review" default). Click Submit.The for this run will now be dark red and the comment will be shown in hover text.
Run Graphs
To see a graphical picture of the run and associated files, click the graph icon.A flow chart of the run is rendered, showing the input data and file outputs. Note that the elements of the flowchart are clickable links to change the focus of the graph.
Switch tabs above the graph for the Graph Details and Text View of the run.
Click Toggle Beta Graph (New!) to see a lineage version.
Return to the main folder by clicking the Folder Name link near the top of the page.
You can return to the runs page by clicking the name of your assay in the Assay List.
Move Runs
Some assay runs grids, including MS2 and Panorama, support moving of runs to another folder via the pipeline. If this option is supported, you can select a run and click Move.The list of available folders will be shown. Only folders configured with a pipeline root will provide links you can click to move the run.
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Assay QC states can be defined for use in approval workflows. Users performing quality control checks can be granted the QC Analyst role and then assign states to runs, indicating progress through a workflow. Assay QC is supported for Standard Assays, the flexible, general purpose type.This topic covers how to use Assay QC states that have been defined and enabled by an administrator. A QC Analyst user is any user assigned this role in addition to their reader or higher role in the container. These QC Analysts can set states on a per run basis.
Assay QC states can vary by workflow. There is always a system "[none]" state, applied to new data by default. The administrator may configure a different default and the states used in your system. In this example, the example QC states defined are "Not Yet Reviewed", "Reviewed - Passed", and "Reviewed - Rejected".
Update QC State
QC Analyst users will see a QC State option on the runs grid for assays where it is enabled. To set (or change) a state for one or more rows:
Viewing the runs grid, select the desired run(s) using checkboxes on the left.
Select QC State > Update state of selected runs.
When you select a single row, you will see the current state and history of state changes, as shown below. When more than one row is selected, you will only see the two entry fields.
New State: (Required) Select the new state for the selected run(s) from the dropdown. You must set this field, even if you are not changing the state from it's current setting.
Comment: Provide a comment about this change; your organization may have requirements for what information is expected here. Note that this field can be configured as either optional or required by an administrator.
Click Update.
Once set, the setting will be shown in the QC Flags column. You can click it to see the associated comment or change the setting for that row.
Display Data Based on State
The visibility of runs, and the corresponding results, is controlled by the QC state setting and whether that state is configured to make the data "public". The "public" state means the data is visible to users who have Reader permission in the container. QC Analyst users can see all the data in a folder.For instance, the default "none" state does not show data to non-analyst users. When an administrator defines states, they elect whether data in that state is "public", meaning accessible to viewers that are granted access to the container.You can further customize displays using the grid view customizer and using the values in the QC Flags column.
Link Data to Study
Assay data can be integrated with related data about subjects under investigation by linking to a study. When Assay QC is in use, assays and studies must be in different folder containers. Learn more about the linking process here: Link Assay Data into a StudyWhen using Assay QC states, data can only be linked to a study if it is in a public state according to the configuration of it's assigned QC state.If a user has admin or QC analyst permissions, and attempts to link data that is not in a public QC state, they will see the error message:
QC checks failed. There are unapproved rows of data in the link to study selection, please change your selection or request a QC Analyst to approve the run data.
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
In cases where assay data is found to be unreliable, such as due to a bad batch of analytes or other issue, the user can exclude specific rows of data from later analysis.This topic covers excluding data for Standard and File-based Assays. Some specialty assay types, such as Luminex, support similar data exclusions.
In order to exclude data, you must have Editor (or higher) permissions in the folder.
Navigate to the Standard or file-based assay data of interest:
In the Assay List web part on the main page of your folder, click the assay name.
Click the AssayID for the run of interest.
Select the row(s) to exclude using the checkboxes.
Click Exclude. (This link is only active when rows are selected.)
Enter a comment about the reason for the exclusion.
Click Confirm.
You will return to viewing the results grid. The excluded rows are still shown, but highlighted to indicate the exclusion.
All exclusion actions are audited and can be viewed by an administrator in the Assay/Experiment events audit log.
View Excluded Data
Excluded data rows are highlighted in the grid itself, and you can view a consolidated report grid of all excluded data.
Click View Excluded Data above the grid.
The Excluded Rows grid shows the original data as well as when and by whom it was excluded, plus the comment entered (if any).
Filter Out Excluded Data
When you expose the "Flagged as Excluded" column, you can filter out this data prior to creating reports and visualizations.
Click View Results or otherwise return to viewing the grid of data where you have made exclusions.
Select (Grid Views) > Customize Grid.
Scroll down and check the box for Flagged as Excluded.
Drag it to where you would like it displayed on the list of Selected Columns.
Click View Grid (or Save as a named grid).
By filtering the values of this column, you can remove excluded data prior to graphing and reporting.
Reimporting after Exclusions
If you reimport assay data after making exclusions, you will be warned in the import UI. You can review the exclusion report using the provided link. There is no option to retain exclusions after the reimport.
Remove Exclusions
To remove an exclusion, it must be removed from the assayExclusion.Exclusions table by a user with access to the schema browser. Users with the site role "Platform Developer" have this access.
Select > Developer Links > Schema Browser.
Select the assayExclusion schema.
Choose the Exclusions table.
Click View Data.
Exclusions are shown in "batches" in this table. All data result rows excluded in one action (with a shared comment) are listed as a single row here. Rather than remove an individual row exclusion, you must remove the entire exclusion action.
Select the exclusion to remove and click (Delete).
After assay data has been imported, you may need to reimport runs from time to time. For instance, an import property may have been incorrectly entered, or a transformation script may have added new calculations which the user would like to run against previously entered data. You cannot simply import the same data file again - previously imported files are remembered by the server to prevent duplicate data - and attempting to import a file again will raise an error.
The reimport process for runs varies by type of instrument data. The general process for a typical assay is outlined in this topic. Different reimport methods are available for the following assays:
Luminex Reimport - When exclusions have been applied, reimporting Luminex runs offers the option to retain the exclusions. A walkthrough of this process is included in the Luminex tutorial. See Reimport Luminex Runs for more information.
Some assay types, including ELISA, ELISpot, and FluoroSpot do not offer a reimport option. If the reimport link or button is not available, the only way to reimport a run is to delete it and import it again from the original source file.
Reimport Assay Data
To reimport a run when the Re-Import Run option is available, follow these steps:
Navigate to the Runs view of the assay data.
Select the run to reimport using the checkbox. In this example, an incorrect "AssayId" value was entered for a run in the general assay tutorial.
Click Re-Import Run. Note that the reimport link will only be active when a single run is checked.
The import process will be run again, generally providing the previously entered import properties as defaults. Change properties as needed, click Next to advance import screens.
The Run Data section offers options on reimport:
Click Show Expected Data Fields to display them for reference.
Click Download Spreadsheet Template to download the expected format.
Choose one of the upload options:
Paste in a tab-separated set of values: Enter new data into the text area.
Upload one or more data files: The previously updated file (or files) for this run will be shown. You can use the previous file(s) or click the button to choose a new file (or files).
Click Save and Finish.
The reimported run has a new rowID and is no longer part of any batch the original run belonged to. Note: if you subsequently delete the reimported run, the original run will be restored.
Note: If you don't see data after reimporting runs, you may have saved the default view of your grid with a run filter. Every time a run is reimported, the run ID is different. To resolve this issue:
Select (Grid Views) > Customize Grid.
Click the Filters tab.
Clear the Row ID filter by clicking the 'X' on it's line.
Click Save, confirm Default grid view for this page is selected.
Click Save again.
Assay Event Logging
When you re-import an assay run, three new assay events are created:
Assay Data Re-imported: This event is attached to the "old run" that is being replaced.
Assay Data Loaded: This event is for the "new run" you import.
Added to Run Group: The new run is added to a new batch, named for the date of the reimport.
Example: Reimport NAb Assay Data
A specific scenario in which Neutralizing Antibody runs must be reimported is to apply alternate curve fits to the data.NAb assay data must be deleted before it can be rerun, and the run may consist of multiple files - a metadata and run data file. NAb assay tools do not offer the general reimport option outlined above. Instead, each Run Details page includes a Delete and Reimport button. The tools remember which file or files were involved in the metadata and run data import.To reimport a NAb run:
Navigate to the NAb Assay Runs list.
Click Run Details for the run to reimport.
Click Delete and Reimport.
The previously entered import properties and files will be offered as defaults. Make changes as needed, then click Next and if needed, make other changes.
Like other data in grids, assay results and run data can be exported in Excel, text, and script formats. In addition, assay run data can be exported into a XAR or set of files, as described in this topic. These exports bundle the data and metadata about the run which can then be imported into another instance or used for analysis in other tools.
When exporting assay runs, the current set of data will be exported, so if any runs have been edited or deleted, the exported archive will contain those changes.
Note that some assay types store additional information in custom tables, which is not included in the Files or XAR export archive. For instance, the Luminex assay prompts the user for information about controls and QC standards which will not be included in these exports.
Export Files
Navigate to the assay dataset of interest and click View Runs.
Click (Export).
Click the Files tab.
Using the radio buttons select either:
Include all files
Include only selected files based on usage in run. Use checkboxes to identify files to include.
Specify the File Name to use. The default is "ExportedRuns.zip".
Click Export.
Export XAR
Navigate to the assay dataset of interest and click View Runs.
Folder Relative (Default): Creates a 'relative' LSID during export. Upon import, the LSID is 'rewritten' so that it will appear to have originated in the local container. This option is essentially creating a new copy of the contents upon import.
Partial Folder Relative: This option preserves the namespace suffix portion of the data and material LSIDs, which is useful when exporting multiple runs at different times (in different XARs) that share a common set of files or materials. Preserving namespace suffixes allows LSIDs to 'reconnect' when runs are imported. In contrast, using the 'Folder Relative' option would cause them to be treated as separate objects in different imports.
Absolute: Writes out the full LSID. This means that after import the LSID will be the same as on the original server. Useful when you are moving the XAR within a server, but can raise errors when the same XAR is imported into two folders on the same server.
Standard assay: A flexible, general purpose assay that can be customized by an administrator to capture any sort of experimental data.
Specialty assays: Built-in assays for certain instruments or data formats, such as ELISA, Luminex, ELIspot, NAb, etc. These have specific data expectations and may offer custom dashboards or analysis behavior.
A developer may define and add a new assay type in a module if required.
Assay design: A specific named instance of an assay type, usually customized to include properties specific to a particular use or project. The design is like a pre-prepared map of how to interpret data imported from the instrument.
Assay run: Import of data from one instrument run using an assay design. Runs are created by researchers and lab technicians who enter values for properties specified in the design.
Assay batch: A set of runs uploaded in a single session. Some properties in a design apply to entire batches of runs.
Assay results or assay data: Individual data elements of the run, for example the intensity of a spot or well.
Assay Types
An assay type corresponds to a class of instrument, provider, or file format. For example, the ELISA assay type provides a basic framework for capturing experimental results from an ELISA instrument.
Assay Designs
An assay design is based on an assay type. When you create an assay design, you start with the basic framework provided by the type and customize it to the specifics of your experiment.When you import instrument data into LabKey server, the assay design describes how to interpret the uploaded data, and what additional input to request about the run.Included in the assay design are:
the column names
the column datatypes (integer, text, etc.)
optional validation or parsing/formatting information
customizable contextual data (also known as "metadata") about your assay, such as who ran the assay, on what instrument, and for which client/project.
To further streamline the process of creating the assay design you need, you can ask LabKey server to infer a best guess design when you upload a representative spreadsheet - then instead of declaring every column from scratch, you might need only edit labels or add non-standard or user-entered metadata fields. The process of inferring an assay design is shown in this tutorial.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
The enzyme linked immunosorbent assay (ELISA) is a plate-based technique that uses a solid-phase enzyme immunoassay (EIA) to detect the presence of a ligand (commonly a protein) in a liquid sample using antibodies directed against the protein to be measured. The layout of controls, samples, and replicates across a 96-well plate is used to evaluate the meaning of data collected from those wells.
Overview
LabKey's ELISA assay type imports and interprets raw data from BioTek Microplate readers. A calibration curve is generated, tracking absorption over concentration. The coefficient of determination is also calculated automatically.Like other built-in plate based assays, each ELISA assay design you create is associated with a specific 96-well plate template. The plate template clarifies what samples and controls are in which wells and is used by the assay design to associate result values in the run data with the correct well groups and specimens.An example application of the ELISA assay is to analyze saliva samples for protein biomarkers that might indicate cancer or autoimmune disease. Samples are applied to the plate at various concentrations. The absorption rate at each concentration is read, plotted, and can be analyzed further to better understand biomarkers.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
The enzyme linked immunosorbent assay (ELISA) is a plate-based technique that uses a solid-phase enzyme immunoassay (EIA) to detect the presence of a ligand (commonly a protein) in a liquid sample using antibodies directed against the protein to be measured. The layout of controls, samples, and replicates across a 96-well plate is used to evaluate the meaning of data collected from those wells.LabKey's ELISA assay type imports and interprets raw data from BioTek Microplate readers.
Log in to your server and navigate to your "Tutorials" project. Create it if necessary.
If you don't already have a server to work on where you can create projects, start here.
If you don't know how to create projects and folders, review this topic.
Create a new subfolder named "ELISA Tutorial". Choose the folder type "Assay".
Next configure a plate template that corresponds to the plate of your ELISA experiment.
In the Assay List web part, click Manage Assays.
Click Configure Plate Templates.
Select "new 96 well (8X12) ELISA default template" from the dropdown.
Click Create to open the editor.
Enter a Template Name, for example, "ELISA Tutorial Plate 1".
Review the shape of the template, clicking the Control, Specimen, and Replicate tabs to see the well groupings. Notice that you could edit the template to match alternate well groupings if needed; instructions can be found in Customize Plate Templates. For the purposes of this tutorial, simply save the default.
When finished, click Save & Close.
Now you can create the assay design using that template:
Click the ELISA Tutorial link near the top of the page to return to the main folder page.
Click New Assay Design.
Click the Specialty Assays tab.
Select ELISA (if it is not already chosen by default and choose "Current Folder (ELISA Tutorial)" from the Assay Location dropdown, then click Choose ELISA Assay.
In the Assay Properties section, enter:
A Name, for example, "ELISA Tutorial Assay".
Select a Plate Template: "ELISA Tutorial Plate 1" (it may already be selected by default).
Review the fields defined in each of the other sections by clicking each heading to page through them. You could add or edit fields, but for this tutorial, leave the default design unchanged.
Click Save.
You will return to the main folder page and see your new design on the assay list.
In the ELISA Tutorial folder's Assay List, click the name you gave your assay design, here: ELISA Tutorial Assay.
Click Import Data.
For Batch Properties, accept the default settings by clicking Next.
In the Run Data field, click Choose File (or Browse) and select one of the sample data files you just downloaded. Make no other changes on this page, then click Next.
On the next page, enter Concentrations for Standard Wells as shown here:
Click Save and Import Another Run.
Repeat the import process for the other two files, selecting the other two Run Data files in turn. The concentrations for standard wells will retain their values so you do not need to reenter them.
Click Save and Finish after the last file.
Visualize the Data
When you are finished importing, you'll see the runs grid showing the files you just uploaded. Since you did not specify any other Assay IDs, the filenames are used. Browse the data and available visualizations:
Hover over the row for Assay Id "biotek_01.xls", then click (details).
You will see a visualization of the Calibration Curve.
Use the Selection options to customize the way you view your data. Learn more about this page in the topic: ELISA Run Details View.
Below the plot, you can click to View Results Grid, which will show you the results data for the specific run as if you had clicked the AssayId instead of the (details) link.
You have now completed the ELISA tutorial. You can continue to explore the options available on the details page in this topic:
When you import ELISA Assay data, you can view details for any run in an interactive visualization panel. By default, we display the standard curve plotted for concentration versus absorption (or raw signal). The calculated concentrations for the sample well groups are plotted as points around the standard curve. The curve is a linear least squares fit and we pass in the parameters of the curve fit (offset and slope).
For a single-plate, low-throughput assay run, you may not have additional selections to make among the following controls. Future development is planned to unlock more functionality in this area. Learn more in Enhanced ELISA Assay Support.
Data Selections
In the Data Selections panel, you can choose:
Samples:
Check the box to Show all Samples. (Default)
Otherwise use the dropdown to select which Samples to plot.
Controls:
Check the box to Show all Controls. (Default)
Otherwise use the dropdown to select which Controls to plot.
Plot Options
X-Axis: The default is a Linear scale, check the box to switch to Log scale.
The default is Concentration. Use the menu to select another metric to plot on this axis:
Y-Axis: The default is a Linear scale, check the box to switch to Log scale.
The default is Absorption. Use the menu to select another metric to plot on this axis.
Display Options:
Show legend: Checked by default. Uncheck to hide it.
Additional Metrics to Plot
For both axes, the set of metrics available include:
Absorption
Absorption CV
Absorption Mean
Concentration
Concentration CV
Concentration Mean
Dilution
Percent Recovery
Percent Recovery Mean
Std Concentration
Curve Fit
Show curve fit line: Checked by default. Uncheck to hide it.
When checked, the panel will display the following for this curve fit:
R Squared
Fit Parameters: Parameters specific to the curve fit.
For Linear curve fits, the Fit Parameters are:
intercept
slope
Calibration Curve
On the right, you'll see the plotted Calibration Curve based on how you've set the above selectors.Below the plot, you have three buttons:
View Results Grid: View the results data for this run.
Export to PNG: Export the current plot display as a PNG file.
Export to PDF: Export the current plot display as a PDF file.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
This topic provides a reference for plate layouts, assay properties, and fields used in the LabKey ELISA Assay implementation. When you design your own plate template and assay design, you can customize and add to these defaults as needed to suit your experiment.
The ELISA Default Template is suitable for an application where you have 5 specimens and a control, typically applying dilution across the wells in the plate. There are three layers, or tabs, in the plate design template: Control, Specimen, and Replicate. The defaults for well layout are as shown:You can change the well layouts, well group names, and add well group properties before saving your new plate template for use.
ELISA Undiluted Template
The ELISA Undiluted Template is suitable for an application where you have up to 42 specimens and a control. Each specimen is present in two wells. There are three layers, or tabs, in the plate design template: Control, Specimen, and Replicate. The defaults for well layout are as shown:You can change the well layouts, well group names, and add well group properties before saving your new plate template for use.
ELISA Assay Properties
The default ELISA assay includes the standard assay properties, except for Editable Results and Import in Background. In addition, ELISA assays use:
Plate Template.
Choose an existing template from the drop-down list.
Edit an existing template or create a new one via the "Configure Templates" button.
ELISA Assay Fields
The default ELISA assay is preconfigured with batch, run, sample, and results fields. You can customize a named design to suit the specifics of your experiment, adding new fields and changing their properties. Learn more in Field Editor.
This feature is under development for a future release of LabKey Server. If you are interested in using this feature, please contact us.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Enhancements to LabKey's ELISA Assay tools are under development, including import of high-throughput data file formats which contain multiple plates and multiple analyte values per well. When completed, the tools will support:
A 4 plex data format. Plates will be able to have up to 4 values per well location, one for each protein/analyte combination.
Additional curve fit options for Standard curves, including 4 and 5 parameter (4pl and 5pl) alongside the existing linear curve fit.
High throughput data support: The current ELISA assay uses a 96 well plate. With these enhancements, 384 well plates are supported.
Additional statistics calculations beyond what is supported currently.
Enable Experimental Feature
To use these enhanced ELISA assay options, a Site Administrator must enable the experimental feature.
Select > Site > Admin Console.
Under Configuration, click Experimental Features.
Under ELISA Multi-plate, multi well data support, click Enable.
Note that experimental features are under development and there is no guarantee that future changes will be backwards compatible with current behavior.
4-plex Data Format
The existing ELISA assay will be extended so that it can import a single file which contains the 4-plex format, in which some of the plates will have 4 values per well location. There will be a row per well/analyte combination. For every row, sample information will also be included and will be parsed out so that sample properties can be stored separately from the plate data.
Additional Curve Fit Options
When using the enhanced ELISA assay functionality, the Run Properties contain an additional field:
CurveFitMethod
This can be set to:
4 Parameter
5 Parameter
Linear
This property may be set by the operator during manual data import, or included in a properly formatted uploaded combined data and metadata file.
High Throughput Plate Template
Before creating any ELISA assay design, create the custom plate templates you require. You can start from our default and make changes as needed to match your experiments. With the experimental features available, in addition to the low-throughput plates supported for ELISA assays, you can now use high-throughput 384 well plates:
Selecting > Manage Assays.
Click Configure Plate Templates.
From the Create New Template dropdown, select new 384 well (16x24) ELISA high-throughput (multi plate) template.
Click Create.
Give your new definition a unique Template Name (even if you make no changes to the defaults).
Use the plate designer to customize the default multiplate layout to suit your needs.
Click Save & Close.
Create ELISA Assay
With the experimental features enabled, you gain the ability to submit a Combined File Upload including metadata alongside your data, instead of requiring the user input metadata manually.
In the Assay List web part, or by selecting > Manage Assays, click New Assay Design.
In the Choose Assay Type panel, click the tab for Specialty Assays.
Under Use Instrument Specific Data Format, select ELISA (if it is not already selected by default).
Under Assay Location, select where you want to define the assay. This selection determines where it will be available.
Click Choose ELISA Assay.
Under Assay Properties, give your new Assay Design a Name.
Select the Plate Template you defined.
Select the Metadata Input Format you will use:
Manual
Combined File Upload (metadata and run data)
Enter other properties as needed.
Review the Batch, Run, Sample, and Results Fields sections, making adjustments as needed.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
The Enzyme-Linked ImmunoSpot (ELISpot) assay is a highly sensitive method for analysis of antigen-specific responses at the cellular level and is widely used to monitor immune responses in humans and other animals. A variety of instruments provide raw ELISpot data, including CTL, Zeiss, and AID. LabKey Server provides a built in ELISpot assay type to support these commonly used assay machines. You can use the built-in assay type, as shown in the tutorial, or you can customize it to your specifications.You can see a sample of detail available for ELIspot results in our interactive example.
The Enzyme-Linked ImmunoSpot (ELISpot) assay is a highly sensitive method for analysis of antigen-specific responses at the cellular level and is widely used to monitor immune responses in humans and other animals. A variety of instruments provide raw ELISpot data, including CTL, Zeiss, and AID. LabKey Server provides a built in ELISpot assay type to support these commonly used assay machines. You can use the built-in assay type, as shown in this tutorial, or you can customize it to your specifications.You can see sample ELISpot data in our interactive example.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
In this first step of the ELISpot tutorial, we create a workspace, design a plate template, create an assay design, and then import some ELISpot data. This tutorial uses the defaults built in to LabKey Server. In actual use, you can customize both template and design to suit your experimental data.
Drag and drop the unzipped LabKeyDemoFiles directory into the upload area of the Files web part.
Configure an ELISpot Plate Template
Select > Manage Assays.
Click Configure Plate Templates.
Select new 96 well (8x12) ELISpot default template from the dropdown.
Click Create to open the Plate Template Editor, which allows you to configure the layouts of specimens and antigens on the plate.
Enter Template name: "Tutorial ELISpot Template"
Explore the template editor. On the Specimen tab, you will see the layout for specimens:
Click the Antigen tab. You will see the layout for antigens.
The Control tab can be used for defining additional well groups. For instance, if you are interested in subtracting background wells, see Background Subtraction for how to define a set of background wells.
For this tutorial, we will simply use the default template. For customization instructions, see Customize Plate Templates.
Click Save & Close.
You now have a new ELISpot plate template that you can use as a basis for creating new assay designs.
Create a New Assay Design Based on the Template
Select > Manage Assays.
Click New Assay Design.
Click the Specialty Assays tab.
Under Use Instrument Specific Data Format, choose ELISpot.
Under Assay Location, choose "Current Folder (ELISpot Tutorial)".
Click Choose ELISpot Assay.
In the Assay Properties section, enter:
Name: "Tutorial ELISpot Design".
Plate Template: choose your new "Tutorial ELISpot Template" if it is not already selected.
Detection Method: choose "colorimetric" if it is not already selected.
Review (but do not change) the fields in the other sections, click the headings to open.
Click Save when finished.
Import ELISpot Runs
Click the ELISpot Tutorial link to return to the main page.
In the Files web part, click the icon to show the folder tree.
Open LabKeyDemoFiles > Assays > Elispot.
Select Zeiss_datafile.txt.
Click Import Data.
In the pop-up, select Use Tutorial ELISpot Design (the one you just created) and click Import.
Batch Properties:
For Participant/Visit, select Specimen/sample id. (Do not check the box for "I will also provide participant id and visit id".)
Click Next.
Run Properties: Enter the following:
AssayID: ES1
Experiment Date: 2009-03-15
Plate Reader: Select "Zeiss" from the pulldown list.
Specimen IDs:
Specimen id values are often barcoded on labels attached to the plates. Enter these sample barcodes (They are taken from the file "LabKeyDemoFiles\Specimens\Specimen Barcodes.pdf" you downloaded earlier. They will integrate with specimens in our demo study.):
526455390.2504.346
249325717.2404.493
249320619.2604.640
249328595.2604.530
Click Next.
Antigen Properties
Fill out the antigen properties according to the screenshot below.
The cells/well applies to all antigens, so you can just fill in the first box in this column with "40000" and click the "Same" checkbox above the column.
The antigen names shown are examples to match our tutorial screenshots; you could enter any values here appropriate to your research.
Click Save and Finish when you are done.
Explore Imported Data
You will see a list of runs for the assay design. See Review ELISpot Data for a walkthrough of the results and description of features for working with this data.
Link Assay Data to the Demo Study (Optional Step)
You can integrate this tutorial ELISpot data into a target study following steps described in the topic: Link Assay Data into a Study. If you have entered matching participant and specimen IDs, you may simply select all rows to link.When the linking is complete, you will see the dataset in the target study. It will look similar to this online example in the demo study on our LabKey Support site.
Click the name of the assay, Tutorial ELISpot Design.
You will see a list of runs for the assay design. (There is only one run in the list at this point.)
Hover over the row and click (Details).
You will see two grids: the data, and a well plate summary.
Note the columns in the first grid include calculated mean and median values for each antigen for each sample well group.
This view is filtered to show only data from the selected run. If you had multiple runs, you could clear that filter to see additional data by clicking the in the filter box above the data section.
The second web part grid represents the ELISpot well plate.
Hover over an individual well to see detailed information about it.
Use the radio buttons to highlight the location of samples and antigens on the plate. Recall the layouts in the plate template we created; the well groups here were defined and associated with certain cells in that template.
Click View Runs to return to the list of runs for our "Tutorial ELISpot Design".
Now click the run name, ES1. If you used the file name, it will read "Zeiss_datafile.txt".
You will see the assay results data.
A similar set of ELISpot assay results may be viewed in the interactive example
Handle TNTC (Too Numerous To Count) Values
ELISpot readers sometimes report special values indicating that a certain spot count in a given well is too numerous to count. Some instruments display the special value -1 to represent this concept, others use the code TNTC. When uploaded ELISpot data includes one of these special values instead of a spot count, the LabKey Server well grid representation will show the TNTC code, and exclude that value from calculations. By essentially ignoring these out of range values, the rest of the results can be imported and calculations done using the rest of the data.If there are too many TNTC values for a given well group, no mean or median will be reported.
Background Subtraction
One option for ELISpot data analysis is to subtract a background value from measured results. When enabled, each specimen group will have a single background mean/median value. Then for each antigen group in the sample, the mean/median for the group is calculated, then the background mean/median is subtracted and the count is normalized by the number of cells per well.
Enable Background Subtraction
To enable background well subtraction, you first configure the plate template. The main flow of the tutorial did not need this setting, but we return there to add it now.
Select > Manage Assays.
Click Configure Plate Templates.
Open the plate template editor for "Tutorial ELISpot Template" by clicking Edit. Or Edit a copy if you prefer to give it a new name and retain the original version.
On the Control tab, create a well group called "Background Wells".
Type the name into the New field, then click Create.
Click Create Multiple to create several groups at once.
Select the wells you want to use as this background group.
Save the plate template.
If you created a new copy of the plate template with subtraction wells, edit the assay design to point to the new plate template.
When an assay design uses a plate with background wells defined, the user can selectively choose background subtraction for imported data by checking the Background Subtraction checkbox during import. When selected, background calculations will be performed. When not selected, or when the plate template does not specify a set of background wells, no background calculations will be performed.On the ELISpot assay runs grid, there is a column displaying whether background subtraction has been performed for a run. The user can select runs in this grid, and then use the Subtract Background button to start a pipeline job to convert existing runs:Once background subtraction calculations have been performed, there is no one-step way to reverse it. Deleting and re-uploading the run without subtraction will achieve this result.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
ELISpot Assays support import of raw data files from Zeiss, CTL, and AID instruments, storing the data in sortable/filterable data grids.The default ELISpot assay type includes some essential properties and fields beyond the defaults included in standard assay designs. You can also add additional properties when you create a new assay design. This topic describes the properties in the default ELISpot assay design and how they are used.
Assay Properties
Assay properties are set by an administrator at the time of assay design and apply to all batches and runs uploaded using that design. The default ELISpot assay includes the standard assay properties, except for Editable Results and Import in Background. In addition, ELISpot assays use:
Plate Template
Choose an existing template from the drop-down list.
Edit an existing template or create a new one via the "Configure Templates" button. For further details, see Customize Plate Templates.
Detection Method: Choose the method used for detection. Options:
colorimetric
fluorescent
Batch Fields
Batch fields receive one value during import of a given batch of runs and apply to all runs in the batch. The default ELISpot assay does not add additional fields to the standard assay type. Data importers will be prompted to enter:
The user is prompted to enter values for run fields which apply to the data in a single file, or run. Run-level fields are stored on a per-well basis, but used for record-keeping and not for calculations.Included by default:
AssayID: The unique name of the run - if not provided, the filename will be used.
Comments
ProtocolName
LabID
PlateID
TemplateID
ExperimentDate
SubtractBackground: Whether to subtract background values, if a background well group is defined. See Background Subtraction for more information.
PlateReader(Required): Select the correct plate reader from the dropdown list. This list is populated with values from the ElispotPlateReader list.
Run Data(Required): Browse or Choose the file containing data for this run.
Sample Fields
For each of the sample/specimen well groups in the chosen plate template, enter the following values in the grid:
SpecimenID: Enter the specimenID for each group here. These values are often barcoded on labels attached to the plates.
SampleDescription: A sample description for each specimen group. If you click the checkbox, all groups will share the same description.
Antigen Fields
The user will be prompted to enter these values for each of the antigen well groups in their chosen plate template, or they will be read from the data file or calculated. Use the Same checkbox to apply the same value to all rows when entering antigen IDs, names, and cells per well.
AntigenID: The integer ID of the antigen.
SpecimenLsid
WellgroupName
Mean
Median
AntigenWellgroupName
AntigenName
Analyte
Cytokine
CellsWell: Cells per well
Analyte Fields
The analyte fields are by default uploaded as part of the data file.
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Flow Cytometry
LabKey Server helps researchers automate high-volume flow cytometry analyses, integrate the results with many kinds of biomedical research data, and securely share both data and analyses. The system is designed to manage large data sets from standardized assays that span many instrument runs and share a common gating strategy. It enables quality control and statistical positivity analysis over data sets that are too large to manage effectively using PC-based solutions.
Manage workflows and quality control in a centralized repository
Export results to Excel or PDF
Securely share any data subset
Build sophisticated queries and reports
Integrate with other experimental data and clinical data
LabKey Server supports the import of flow data from FlowJo, a popular flow analysis tool.
An investigator first defines a gate
template for an entire study using FlowJo, and uploads the FlowJo
workspace to LabKey. He or she then points LabKey Flow to a
repository of FCS files.Once the data has been imported, LabKey Server starts an analysis, computes the compensation matrix, applies gates, calculates statistics, and generates graphs. Results are stored in a relational database and displayed using secure, interactive web pages.Researchers can define custom queries and views to analyze large result sets.
Gate templates can be modified, and new analyses can be run and
compared.
To get started, see the introductory flow tutorial: Tutorial: Explore a Flow Workspace
Flow Cytometry Overview
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
LabKey Server enables high-throughput analysis for several types of assays, including flow cytometry assays. LabKey’s flow cytometry solution provides a high-throughput pipeline for processing flow data. In addition, it delivers a flexible repository for data, analyses and results.
This page reviews the FlowJo-only approach for analyzing smaller quantities of flow data, then explains the two ways LabKey Server can help your team manage larger volumes of data. It also covers LabKey Server’s latest enhancement (tracking of background well information) and future enhancements to the LabKey Flow toolkit.
Overview
Challenges of Using FlowJo Alone
Traditionally, analysis of flow cytometry data begins with the download of FCS files from a flow cytometer. Once these files are saved to a network share, a technician loads the FCS files into a new FlowJo workspace, draws a gating hierarchy and adds statistics. The product of this work is a set of graphs and statistics used for further downstream analysis. This process continues for multiple plates. When analysis of the next plate of samples is complete, the technician loads the new set of FCS files into the same workspace.Moderate volumes of data can be analyzed successfully using FlowJo alone; however, scaling up can prove challenging. As more samples are added to the workspace, the analysis process described above becomes quite slow. Saving separate sets of sample runs into separate workspaces does not provide a good solution because it is difficult to manage the same analysis across multiple workspaces. Additionally, looking at graphs and statistics for all the samples becomes increasingly difficult as more samples are added.
Solutions: Using LabKey Server to Scale Up
LabKey Server can help you scale up your data analysis process in two ways: by streamlining data processing and by serving as a flexible data repository. When your data are relatively homogeneous, you can use your LabKey Server to apply an analysis script generated by FlowJo to multiple runs. When your data are too heterogeneous for analysis by a single script, you can use your LabKey Server as a flexible data repository for large numbers of analyses generated by FlowJo workspaces. Both of these options help you speed up and consolidate your work.
Use LabKey as a Data Repository for FlowJo Analyses
Analyses performed using FlowJo can be imported into LabKey where they can be refined, quality controlled, and integrated with other related data. The statistics calculated by FlowJo are read upon import from the workspace.Graphs are generated for each sample and saved into the database. Note that graphs shown in LabKey are recreated from the Flow data using a different analysis engine than FlowJo uses. They are intended to give a rough 'gut check' of accuracy of the data and gating applied, but are not straight file copies of the graphs in FlowJo.
Annotate Flow Data Using Metadata, Keywords, and Background Information
Extra information can be linked to the run after the run has been imported via either LabKey Flow or FlowJo. Sample information uploaded from an Excel spreadsheet can also be joined to the well. Background wells can then be used to subtract background values from sample wells. Information on background wells is supplied through metadata.
The LabKey Flow Dashboard
You can use LabKey Server exclusively as a data repository and import results directly from a FlowJo workspace, or create an analysis script from a FlowJo workspace to apply to multiple runs. The dashboard will present relevant tasks and summaries.
LabKey Flow Module
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
The LabKey Flow module automates high-volume flow cytometry analysis. It is designed to manage large data sets from standardized assays spanning many instrument runs that share a common gating strategy.To begin using LabKey Flow, an investigator first defines a gate template for an entire study using FlowJo, and uploads the FlowJo workspace to LabKey Server. He or she then points LabKey Flow to a repository of FCS files on a network file server, and starts an analysis.LabKey Flow computes the compensation matrix, applies gates, calculates statistics, and generates graphs. Results are stored in a relational database and displayed using secure, interactive web pages.Researchers can then define custom queries and views to analyze large result sets. Gate templates can be modified, and new analyses can be run and compared. Results can be printed, emailed, or exported to tools such as Excel or R for further analysis. LabKey Flow enables quality control and statistical positivity analysis over data sets that are too large to manage effectively using PC-based solutions.
Note that graphs shown in LabKey are recreated from the Flow data using a different analysis engine than FlowJo uses. They are intended to give a rough 'gut check' of accuracy of the data and gating applied.
If you are using FlowJo version 10.8, you will need to upgrade to LabKey Server version 21.7.3 (or later) in order to properly handle "NaN" (Not a Number) in statistic values. Contact your Account Manager for details.
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Follow the steps in this topic to set up a "Flow Tutorial" folder. You will set up and import a basic workspace that you can use for two of the flow tutorials, listed at the bottom of this topic.
In the Flow Tutorial folder, under the Flow Summary section, click Upload and Import.
Drag and drop the unzipped DemoFlow folder into the file browser. Wait for the sample files to be uploaded.
When complete, you will see the files added to your project's file management system.
Import FlowJo Workspace
More details and options for importing workspaces are covered in the topic: Import a Flow Workspace and Analysis. To simply set up your environment for a tutorial, follow these steps:
Click the Flow Tutorial link to return to the main folder page.
Click Import FlowJo Workspace Analysis.
Click Browse the pipeline and select DemoFlow > ADemoWorkspace.wsp as shown:
Click Next.
Wizard Acceleration
When the files are in the same folder as the workspace file and all files are already associated with samples present in the workspace, you will skip the next two panels of the import wizard. If you need to adjust anything about either step, you can use the 'Back' option after the acceleration. Learn about completing these steps in this topic: Import a Flow Workspace and Analysis.
On the Analysis Folder step, click Next to accept the defaults.
Click Finish at the end of the wizard.
Wait for the import to complete, during which you will see status messages.
When complete, you will see the workspace data.
Click the Flow Tutorial link to return to the main folder page.
You are now set up to explore the Flow features and tutorials in your folder.
Note there is an alternative way to kick off the data import: in the File Repository, select the .wsp file (in this tutorial ADemoWorkspace.wsp), and click Import FlowJo Workspace. Then complete the rest of the import wizard as described above.
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Flow Dashboard Overview
The main Flow dashboard displays the following web parts by default:
Flow Experiment Management: Links to perform actions like setting up an experiment and analyzing FCS files.
Flow Summary: Common actions and configurations.
Flow Analyses: Lists the flow analyses that have been performed in this folder.
Flow Scripts: Lists analysis scripts. An analysis script stores the gating template definition, rules for calculating the compensation matrix, and the list of statistics and graphs to generate for an analysis.
Understanding Flow Column Names
In the flow workspace, statistics column names are of the form "subset:stat". For example, "Lv/L:%P" is used for the "Live Lymphocytes" subset and the "percent of parent" statistic.Graphs are listed with parentheses and are of the form "subset(x-axis:y-axis)". For example, "Singlets(FSC-A:SSC-A)" for the "Singlets" subset showing forward and side scatter.
Customize Your Grid View
The columns displayed by default for a dataset are not necessarily the ones you are most interested in, so you can customize which columns are included in the default grid. See Customize Grid Views for general information about customizing grids.In this first tutorial step, we'll show how you might remove one column, add another, and save this as the new default grid. This topic also explains the column naming used in this sample flow workspace.
Begin on the main page of your Flow Tutorial folder.
Click Analysis in the Flow Analyses web part.
Click the Name: ADemoWorkspace.wsp.
Select (Grid Views) > Customize Grid.
This grid shows checkboxes for "Available Fields" (columns) in a panel on the left. The "Selected Fields" list on the right shows the checked fields.
In the Selected Fields pane, hover over "Compensation Matrix". Notice that you see a tooltip with more information about the field and the (rename) and (remove) icons are activated.
Click the to remove the field.
In the Available Fields pane, open the Statistic node by clicking the icon (it will turn into a ) and click the checkbox for Singlets:%P to add it to the grid.
By default, the new field will be listed after the field that was "active" before it was added - in this case "Count" which was immediately after the removed field. You can drag and drop the field elsewhere if desired.
Before leaving the grid customizer, scroll down through the "Selected Fields" and notice some fields containing () parentheses. These are graph fields. You can also open the "Graph" node in the available fields panel and see they are checked.
Confirm that Default grid view for this page is selected, and click Save.
You will now see the "Compensation Matrix" column is gone, and the "Singlets:%P" column is shown in the grid.Notice that the graph columns listed as "selected" in the grid customizer are not shown as columns. The next step will cover displaying graphs.
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
In this step we will examine our data graphs. As you saw in the previous step, graphs are selected within the grid customizer but are not shown by default.
Note that graphs shown in LabKey are recreated from the Flow data using a different analysis engine than FlowJo uses. They are intended to give a rough 'gut check' of accuracy of the data and gating applied, but may show differences from the original graphs.
Review Graphs
Return to the grid you customized (if you navigated away): from the Flow Tutorial page, in the Flow Analyses web part, click Analysis then ADemoWorkspace.wsp to return to the grid.
Select Show Graphs > Inline.
The inline graphs are rendered. Remember we had selected 4 graphs; each row of the data grid now has an accompanying set of 4 graphs interleaved between them.
Note: for large datasets, it may take some time for all graphs to render. Some metrics may not have graphs for every row.
Three graph size options are available at the top of the data table. The default is medium.
When viewing medium or small graphs, clicking on any graph image will show it in the large size.
See thumbnail graphs in columns alongside other data by selecting Show Graphs > Thumbnail.
Hovering over a thumbnail graph will pop forward the same large format as in the inline view.
The following pages provide other views and visualizations of the flow data.
Scroll down to the bottom of the ADemoWorkspace.wsp page.
Click Show Compensation to view the compensation matrix. In this example workspace, the graphs are not available, but you can see what compensated and uncompensated graphs would look like in a different dataset by clicking here.
Go back in the browser.
Click Experiment Run Graph and then choose the tab Graph Detail View to see a graphical version of this experiment run.
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Before you export your dataset, customize your grid to show the columns you want to export. For greater control of the columns included in a view, you can also create custom queries. Topics available to assist you:
After you have finalized your grid, you can export the displayed table to an Excel spreadsheet, a text file, a script, or as an analysis. Here we show an Excel export. Learn about exporting in an analysis archive in this topic:
Choose the desired format using the tabs, then select options relevant to the format. For this tutorial example, select Excel (the default) and leave the default workbook selected.
Click Export.
Note that export directly to Excel limits the number of rows. If you need to work around this limitation to export larger datasets, first export to a text file, then open the text file in Excel.
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Quality control reports in the flow module can give you detailed insights into the statistics, data, and performance of your flow data, helping you to spot problems and find solutions. Monitoring controls within specified standard deviations can be done with Levey-Jennings plots. To generate a report:
Click the Flow Tutorial link to return to the main folder page.
Click Create QC Report in the new Flow Reports web part.
Provide a Name for the report, and select a Statistic.
Choose whether to report on Count or Frequency of Parent.
In the Filters box, you can filter the report by keyword, statistic, field, folder, and date.
Click Save.
The Flow Reports panel will list your report.
Click the name to see the report. Use the manage link for the report to edit, copy, delete, or execute it.
The report displays results over time, followed by a Levey-Jennings plot with standard deviation guide marks. TSV format output is shown below the plots.
Congratulations
You have now completed the Tutorial: Explore a Flow Workspace. To explore more options using this same sample workspace, try this tutorial next:
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
If you have a flow cytometry experiment containing a background control (such as an unstimulated sample, an unstained sample, or a Fluorescence Minus One (FMO) control), you can set the background in LabKey for use in analysis. To perform this step we need to:
Identify the background controls
Associate the background values with their corresponding samples of interest
This tutorial shows you how to associate sample descriptions with flow data, define and set new keywords, and set metadata including background information. Positive and negative responses can then be calculated against this background.To gain familiarity with the basics of the flow dashboard, data grids, and using graphs, you can complete the Tutorial: Explore a Flow Workspace first. It uses the same setup and sample data as this tutorial.
Sample descriptions give the flow module information about how to interpret a group of FCS files using keywords. The flow module uses a sample type named "Samples" which you must first define and structure correctly. By default it will be created at the project level, and thus shared with all other folders in the project, but there are several options for sharing sample definitions. Note that it is important not to rename this sample type. If you notice "missing" samples or are prompted to define it again, locate the originally created sample type and restore the "Samples" name.Download these two files to use:
Check that the fields added match the screencap below, and adjust names and types if not.
Learn more about the field editor in this topic: Field Editor.
Ignore any banner offering to add a unique ID field.
Click Save.
Return to the main dashboard by clicking the Flow Tutorial link near the top of the page.
You have now defined the sample type you need and will upload the actual sample information next.
Upload Samples
In the Flow Tutorial folder, click Upload Samples.
You can either copy/paste data as text or upload a file. For this tutorial, click Upload File.
Leave the default Add samples selected.
Click Browse to locate and select the "FlowSampleDemo.xlsx" file you downloaded.
Do not check the checkbox for importing by alternate key.
Click Submit.
Once it is uploaded, you will see the sample type.
Associate the Sample Descriptions with the FCS Files
Return to the main dashboard by clicking the Flow Tutorial link near the top of the page.
Click Define Sample Description Join Fields in the Flow Experiment Management web part.
On the Join Samples page, set how the properties of the sample type should be mapped to the keywords in the FCS files.
For this tutorial, select TubeName from the Sample Property menu, and Name from the FCS Property menu
Click Update.
You will see how many FCS Files could be linked to samples. The number of files linked to samples may be different from the number of sample descriptions since multiple files can be linked to any given sample.To see the grid of which samples were linked to which files:
Click the Flow Tutorial link.
Then click 17 sample descriptions (under "Assign additional meanings to keywords").
Linked Samples and FCSFiles are shown first, with the values used to link them.
Scroll down to see Unlinked Samples and Unlinked FCSFiles, if any.
Add Keywords as Needed
Keywords imported from FlowJo can be used to link samples, as shown above. If you want to add additional information within LabKey, you can do so using additional keywords.
Return to the main Flow Tutorial dashboard.
Click 17 FCS files under Import FCS Files.
Select the rows to which you want to add a keyword and set a value. For this tutorial, click the box at the top of the column to select all rows.
Click Edit Keywords.
You will see the list of existing keywords.
The blank boxes in the righthand column provide an opportunity to change the values of existing keywords. All selected rows would receive any value you assigned a keyword here. If you leave the boxes blank, no changes will be made to the existing data.
For this tutorial, make no changes to the existing keyword values.
Click Create a new keyword to add a new keyword for the selected rows.
Enter "SampleType" as the keyword, and "Blood" as the value.
Click Update.
You will see the new SampleType column in the files data grid, with all (selected) rows set to "Blood".
If some of the samples were of another type, you could repeat the "Edit Keywords" process for the subset of rows of that other type, entering the same "SampleType" keyword and the alternate value.
Set Metadata
Setting metadata including participant and visit information for samples makes it possible to integrate flow data with other data about those participants.Background and foreground match columns are used to identify the group -- using subject and timepoint is one option.Background setting: it is used to identify which well or wells are background out of the group of wells. If more than one well is identified as background in the group, the background value will be averaged.
Click the Flow Tutorial link near the top of the page to return to the main flow dashboard.
Click Edit Metadata in the Flow Summary web part.
On the Edit Metadata page, you can set three categories of metadata. Be sure to click Set Metadata at the bottom of the page when you are finished making changes.
Sample Columns: Select how to identify your samples. There are two options:
Set the Specimen ID columnor
Set the Participant columnand set either a Visit columnorDate column.
Background and Foreground Match Columns
Select the columns that match between both the foreground and background wells.
You can select columns from the uploaded data, sample type, and keywords simultaneously as needed. Here we choose the participant ID since there is a distinct baseline for each participant we want to use.
Background Column and Value
Specify the column and how to filter the values to uniquely identify the background wells from the foreground wells.
Here, we choose the "Sample WellType" column, and declare rows where the well type "Contains" "Baseline" as the background.
When finished, click Set Metadata.
Use Background Information
Click the Flow Tutorial link near the top of the page to return to the main flow dashboard.
Click the Analysis Folder you want to view, here Analysis (1 run), in the Flow Summary web part.
Click the name ADemoWorkspace.wsp to open the grid of workspace data.
Control which columns are visible using the standard grid customization interface, which will include a node for background values once the metadata is set.
Select (Grid Views) > Customize Grid.
Now you will see a node for Background.
Click the icon to expand it.
Use the checkboxes to control which columns are displayed.
If they are not already checked, place checkmarks next to "BG Count", "BG Live:Count", "BG Live:%P", and "BG Singlets:Count".
Click Save, then Save in the popup to save this grid view as the default.
Return to the view customizer to make the source of background data visible:
Select (Grid Views) > Customize Grid.
Expand the FCSFile node, then the Sample node, and check the boxes for "PTID" and "Well Type".
Click View Grid.
Click the header for PTID and select Sort Ascending.
Click Save above the grid, then Save in the popup to save this grid view as the default.
Now you can see that the data for each PTID's "Baseline" well is entered into the corresponding BG fields for all other wells for that participant.
Display graphs by selecting Show Graphs > Inline as shown here:
Congratulations
You have now completed the tutorial and can use this process with your own data to analyze data against your own defined background.
Import a Flow Workspace and Analysis
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Once you have set up the folder and uploaded the FCS files, you can import a FlowJo workspace and then use LabKey Server to extract data and statistics of interest.This topic uses a similar example workspace as in the Set Up a Flow Folder walkthrough, but includes intentional mismatches to demonstrate the full import wizard in more detail.
In the Flow Tutorial folder, under the Flow Summary section, click Upload and Import.
Click fileset in the left panel to ensure you are at the 'top' level of the file repository
Drop the unzipped DemoFlowWizard folder into the target area to upload it.
Click the Flow Tutorial link to return to the main folder page.
Click Import FlowJo Workspace Analysis. This will start the process of importing the compensation and analysis (the calculated statistics) from a FlowJo workspace. The steps are numbered and the active step will be shown in bold.
1. Select Workspace
Select Browse the pipeline.
In the left panel, click to expand the DemoFlowWizard folder.
In the right panel, select the ADemoWorkspace.wsp file, and click Next.
Warnings
If any calculations are missing in your FlowJo workspace, you would see warnings at this point. They might look something like:
The tutorial does not contain any such warnings, but if you did see them with your own data and needed to import these statistics, you would have to go back to FlowJo, re-calculate the missing statistics, and then save as xml again.2. Select FCS Files
If you completed the flow tutorial, you may have experienced an accelerated import wizard which skipped steps. If the wizard cannot be accelerated, you will see a message indicating the reason. In this case, the demo files package includes an additional file that is not included in the workspace file. You can proceed, or cancel and make adjustments if you expected a 1:1 match.To proceed:
If you have already imported the tutorial files into this folder, choose Previously imported FCS files in order to see the sample review process in the next step.
If not, select Browse the pipeline for the directory of FCS files and check the box for the DemoFlowWizard folder.
Click Next.
3. Review Samples
The import wizard will attempt to match the imported samples from the FlowJo workspace with the FCS files. If you were importing samples that matched existing FCS files, such as reimporting a workspace, matched samples would have a green checkmark and unmatched samples would have a red checkmark. To manually correct any mistakes, select the appropriate FCS file from the combobox in the Matched FCS File column. See below for more on the exact algorithm used to resolve the FCS files.
Confirm that all samples (= 17 samples) are selected and click Next.
4. Analysis Folder
Accept the default name of your analysis folder, "Analysis". If you are analyzing the same set of files a second time, the default here will be "Analysis1".
Click Next.
5. Confirm
Review the properties and click Finish to import the workspace.
Wait for Import to Complete. While the job runs, you will see the current status file growing and have the opportunity to cancel if necessary. Import can take several minutes.
When the import process completes, you will see a grid named "ADemoWorkspace.wsp" To learn how to customize this grid to display the columns of your choice, see this topic: Step 1: Customize Your Grid View.
Resolving FCS Files During Import
When importing analysis results from a FlowJo workspace or an external analysis archive, the Flow Module will attempt to find a previously imported FCS file to link the analysis results to.The matching algorithm compares the imported sample from the FlowJo workspace or external analysis archive against previously imported FCS files using the following properties and keywords: FCS file name or FlowJo sample name, $FIL, GUID, $TOT, $PAR, $DATE, $ETIM. Each of the 7 comparisons are weighted equally. Currently, the minimum number of required matches is 2 -- for example, if only $FIL matches and others don't, there is no match.While calculating the comparisons for each imported sample, the highest number of matching comparisons is remembered. Once complete, if there is only a single FCS file that has the max number of matching comparisons, it is considered a perfect match. The import wizard resolver step will automatically select the perfectly matching FCS file for the imported sample (they will have the green checkmark). As long as each FCS file can be uniquely matched by at least two comparisons (e.g., GUID and the other keywords), the import wizard should automatically select the correct FCS files that were previously imported.If there are no exact matches, the imported sample will not be automatically selected (red X mark in the wizard) and the partially matching FCS files will be listed in the combo box ordered by number of matches.
Name Length Limitation
The names of Statistics and Graphs in the imported workspace cannot be longer than 400 characters. FlowJo may support longer names, but they cannot be imported into the LabKey Flow module. Names that exceed this limit will generate an import error similar to:
11 Jun 2021 16:51:21,656 ERROR: FlowJo Workspace import failed org.labkey.api.query.RuntimeValidationException: name: Value is too longfor column 'SomeName', a maximum length of 400 is allowed. Supplied value was 433 characters long.
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Keywords imported from FlowJo can be used to link samples and provide metadata. This topic describes how you can edit the values assigned to those keywords and also add additional keywords within LabKey to store additional information.
You can edit keyword input values either individually or in bulk.
To edit existing FCS keywords, go to the FCS Files table and select one or more rows. Any changes made will be applied to all the selected rows.
Enter the new values for desired keywords in the blank text boxes provided.
The FCS files to be changed are listed next to Selected Files.
Existing values are not shown in the input form. Instead, a blank form is shown, even for keywords that have non-blank values.
Click Update to submit the new values.
If multiple FCS files are selected, the changes will be applied to them all.
If an input value is left blank, no change will be made to the existing value(s) of that keyword for the selected rows.
Add Keywords and Set Values
To add a new keyword, click Create a New Keyword
Enter the name of the keyword and a value for the selected rows. Both keyword and value are required.
Note that if you want to add a new keyword for all rows, but set different values, you would perform multiple rounds of edits. First select all rows to add the new keyword, providing a default/common temporary value for all rows. Then select the subsets of rows with other values for that keyword and set a new value.
Display Keyword Values
New keywords are not always automatically shown in the data grid. To add the new keyword column to the grid:
Select (Grid Views) > Customize Grid.
In the Available Fields panel, open the node Keywords and select the new keyword.
The flow module uses a sample type named "Samples". You cannot change this name expectation, and the type's properties and fields must be defined and available in your folder before you can upload or link to sample descriptions. Each folder container could have a different definition of the "Samples" sample type, or the definition could be shared at the project or site level.
Note that it is important not to rename this sample type later. If you notice "missing" samples or are prompted to define it again, locate the originally created sample type and restore the "Samples" name.
For example, you might have a given set of samples and need to run a series of flow panels against the same samples in a series of subfolders.
Or you might always have samples with the same properties across a site, though each project folder has a unique set of samples.
Site-wide Sharing
If you define a "Samples" sample type in the Shared project, you will be able to use the definition in any folder on the site. Each folder will have a distinct set of samples local to the container, i.e. any samples themselves defined in the Shared project will not be exposed in any local flow folder.Follow the steps below to:
If you define a "Samples" sample type in the top level project, you will be able to use the definition in all subfolders of that project at any level. Each folder will also be able to share the samples defined in the project, i.e. you won't have to import sample descriptions into the local flow folder.Follow the steps below to:
Note that a "Samples" sample type defined in a folder will not be shared into subfolders. Only project-level definitions and samples are "inherited" by subfolders.
Create "Samples" Sample Type
To create a "Samples" sample type in a folder where it does not already exist (and where you do not want to share the definition), follow these steps.
From the flow dashboard, click either Upload Sample Descriptions or Upload Samples.
If you don't see the link for "Upload Sample Descriptions", and instead see "Upload More Samples", the sample type is already defined and you can proceed to uploading.
On the Create Sample Type page, you cannot change the Name; it must be "Samples" in the flow module.
You need to provide a unique way to identify the samples of this type.
If your sample data contains a Name column, it will be used as the unique identifier.
If not, you can provide a Naming Pattern for uniquely identifying the samples. For example, if you have a "TubeName" column, you can tell the server to use it by making the Naming Pattern "${TubeName}" as shown here:
Click the Fields section to open it.
Review the Default System Fields.
Fields with active checkboxes in the Enabled column can be 'disabled' by unchecking the box.
The Name of a sample is always required, and you can check the Required boxes to make other fields mandatory.
Collapse this section by clicking the . It will become a .
You have several options for defining Custom Fields for your samples:
Import or infer fields from file: Drag and drop or select a file to use for defining fields:
Infer from a data spreadsheet. Supported formats include: .csv, .tsv, .txt, .xls, and .xlsx.
When finished, click Save to create the sample type.
You will return to the main dashboard where the link Upload Sample Descriptions now reads Upload More Samples.
Upload Sample Descriptions
Once the "Samples" sample type is defined, clicking Upload Samples (or Upload More Samples) will open the panel for importing your sample data.
Click Download Template to download a template showing all the necessary columns for your sample type.
You have two upload options:
Either Copy/Paste data into the Data box shown by default, and select the appropriate Format: (TSV or CSV). The first row must contain column names.
OR
Click Upload file (.xlsx, .xls, .csv, .txt) to open the file selection panel, browse to your spreadsheet and open it. For our field definitions, you can use this spreadsheet: FlowSampleDemo.xlsx
Import Options: The default selection is Add Samples, i.e. fail if any provided samples already defined.
Change the selection if you want to Update samples, and check the box if you also want to Allow new samples during update, otherwise, new samples will cause import to fail.
Once the samples are defined, either by uploading locally, or at the project-level, you can associate the samples with FCS files using one or more sample join fields. These are properties of the sample that need to match keywords of the FCS files.
Click Flow Tutorial (or other name of your folder) to return to the flow dashboard.
Click Define sample description join fields and specify the join fields. Once fields are specified, the link will read "Modify sample description join fields". For example:
Sample Property
FCS Property
"TubeName"
"Name"
Click Update.
Return to the flow dashboard and click the link ## sample descriptions (under "Assign additional meanings to keywords"). You will see which Samples and FCSFiles could be linked, as well as the values used to link them.Scroll down for sections of unlinked samples and/or files, if any. Reviewing the values for any entries here can help troubleshoot any unexpected failures to link and identify the right join fields to use.
Add Sample Columns to View
You will now see a new Sample column in the FCSFile table and can add it to your view:
Click Flow Tutorial (or other name of your folder).
Click FCS Files in the Flow Summary on the right.
Click the name of your files folder (such as DemoFlow).
Select (Grid Views) > Customize Grid.
Expand the Sample node to see the columns from this table that you may add to your grid.
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
This topic covers information helpful in writing some flow-specific queries. Users who are new to custom queries should start with this section of the documentation:
For this example, we create a query called "StatisticDemo" based on the FCSAnalyses dataset. Start from your Flow demo folder, such as that created during the Flow Tutorial.
Create a New Query
Select > Go To Module > Query.
Click flow to open the flow schema.
Click Create New Query.
Call your new query "StatisticDemo"
Select FCSAnalyses as the base for your new query.
Click Create and Edit Source.
Add Statistics to the Generated SQL
The default SQL simply selects all the columns:
SELECT FCSAnalyses.Name, FCSAnalyses.Flag, FCSAnalyses.Run, FCSAnalyses.CompensationMatrix FROM FCSAnalyses
Add a line to include the 'Count' statistic like this. Remember to add the comma to the prior line.
SELECT FCSAnalyses.Name, FCSAnalyses.Flag, FCSAnalyses.Run, FCSAnalyses.CompensationMatrix, FCSAnalyses.Statistic."Count" FROM FCSAnalyses
Click Save.
Click the Data tab. The "Count" statistic has been calculated using the Statistic method on the FCSAnalyses table, and is shown on the right.
You can flip back and forth between the source, data, and xml metadata for this query using the tabs in the query editor.
Run the Query
From the "Source" tab, to see the generated query, either view the "Data" tab, or click Execute Query. To leave the query editor, click Save & Finish.The resulting table includes the "Count" column on the right:View this query applied to a more complex dataset. The dataset used in the Flow Tutorial has been slimmed down for ease of use. A larger, more complex dataset can be seen in this table:
Example: SubsetDemo Query
It is possible to calculate a suite of statistics for every well in an FCS file using an INNER JOIN technique in conjunction with the "Statistic" method. This technique can be complex, so we present an example to provide an introduction to what is possible.
SELECT FCSAnalyses.FCSFile.Run AS ASSAYID, FCSAnalyses.FCSFile.Sample AS Sample, FCSAnalyses.FCSFile.Sample.Property.PTID, FCSAnalyses.FCSFile.Keyword."WELL ID" AS WELL_ID, FCSAnalyses.Statistic."Count" AS COLLECTCT, FCSAnalyses.Statistic."S:Count" AS SINGLETCT, FCSAnalyses.Statistic."S/Lv:Count" AS LIVECT, FCSAnalyses.Statistic."S/Lv/L:Count" AS LYMPHCT, FCSAnalyses.Statistic."S/Lv/L/3+:Count" AS CD3CT, Subsets.TCELLSUB, FCSAnalyses.Statistic(Subsets.STAT_TCELLSUB) AS NSUB, FCSAnalyses.FCSFile.Keyword.Stim AS ANTIGEN, Subsets.CYTOKINE, FCSAnalyses.Statistic(Subsets.STAT_CYTNUM) AS CYTNUM, FROM FCSAnalyses INNER JOIN lists.ICS3Cytokine AS Subsets ON Subsets.PFD IS NOT NULL WHERE FCSAnalyses.FCSFile.Keyword."Sample Order" NOT IN ('PBS','Comp')
Examine the Query
This SQL code leverages the FCSAnalyses table and a list of desired statistics to calculate those statistics for every well.The "Subsets" table in this query comes from a user-created list called "ICS3Cytokine" in the Flow Demo. It contains the group of statistics we wish to calculate for every well.
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
LabKey modules expose their data to the LabKey query engine in one or more schemas. This reference topic outlines the schema used by the Flow module to assist you when writing custom Flow queries.
Flow Module
The Flow schema has the following tables in it:
Runs Table
This table shows experiment runs for all three of the Flow protocol steps. It has the following columns:
RowId
A unique identifier for the run. Also, when this column is used in a query, it is a lookup back to the same row in the Runs table. That is, including this column in a query will allow the user to display columns from the Runs table that have not been explicitly SELECTed into the query
Flag
The flag column. It is displayed as an icon which the user can use to add a comment to this run. The flag column is a lookup to a table which has a text column “comment”. The icon appears different depending on whether the comment is null.
Name
The name of the run. In flow, the name of the run is always the name of the directory which the FCS files were found in.
Created
The date that this run was created.
CreatedBy
The user who created this run.
Folder
The folder or project in which this run is stored.
FilePathRoot
(hidden) The directory on the server's file system where this run's data files come from.
LSID
The life sciences identifier for this run.
ProtocolStep
The flow protocol step of this run. One of “keywords”, “compensation”, or “analysis”
RunGroups
A unique ID for this run.
AnalysisScript
The AnalysisScript that was used in this run. It is a lookup to the AnalysisScripts table. It will be null if the protocol step is “keywords”
Workspace
CompensationMatrix
The compensation matrix that was used in this run. It is a lookup to the CompensationMatrices table.
TargetStudy
WellCount
The number of FCSFiles that we either inputs or outputs of this run.
FCSFileCount
CompensationControlCount
FCSAnalysisCount
CompensationMatrices Table
This table shows all of the compensation matrices that have either been calculated in a compensation protocol step, or uploaded. It has the following columns in it:
RowId
A unique identifier for the compensation matrix.
Name
The name of the compensation matrix. Compensation matrices have the same name as the run which created them. Uploaded compensation matrices have a user-assigned name.
Flag
A flag column to allow the user to add a comment to this compensation matrix
Created
The date the compensation matrix was created or uploaded.
Protocol
(hidden) The protocol that was used to create this compensation matrix. This will be null for uploaded compensation matrices. For calculated compensation matrices, it will be the child protocol “Compensation”
Run
The run which created this compensation matrix. This will be null for uploaded compensation matrices.
Value
A column set with the values of compensation matrix. Compensation matrix values have names which are of the form “spill(channel1:channel2)”
In addition, the CompensationMatrices table defines a method Value which returns the corresponding spill value.
The Value method would be used when the name of the statistic is not known when the QueryDefinition is created, but is found in some other place (such as a table with a list of spill values that should be displayed).
FCSFiles Table
The FCSFiles table lists all of the FCS files in the folder. It has the following columns:
RowId
A unique identifier for the FCS file
Name
The name of the FCS file in the file system.
Flag
A flag column for the user to add a comment to this FCS file on the server.
Created
The date that this FCS file was loaded onto the server. This is unrelated to the date of the FCS file in the file system.
Protocol
(hidden) The protocol step that created this FCS file. It will always be the Keywords child protocol.
Run
The experiment run that this FCS file belongs to. It is a lookup to the Runs table.
Keyword
A column set for the keyword values. Keyword names are case sensitive. Keywords which are not present are null.
Sample
The sample description which is linked to this FCS file. If the user has not uploaded sample descriptions (i.e. defined the target table), this column will be hidden. This column is a lookup to the samples.Samples table.
In addition, the FCSFiles table defines a method Keyword which can be used to return a keyword value where the keyword name is determined at runtime.
FCSAnalyses Table
The FCSAnalyses table lists all of the analyses of FCS files. It has the following columns:
RowId
A unique identifier for the FCSAnalysis
Name
The name of the FCSAnalysis. The name of an FCSAnalysis defaults to the same name as the FCSFile. This is a setting which may be changed.
Flag
A flag column for the user to add a comment to this FCSAnalysis.
Created
The date that this FCSAnalysis was created.
Protocol
(hidden) The protocol step that created this FCSAnalysis. It will always be the Analysis child protocol.
Run
The run that this FCSAnalysis belongs to. Note that FCSAnalyses.Run and FCSAnalyses.FCSFile.Run refer to different runs.
Statistic
A column set for statistics that were calculated for this FCSAnalysis.
Graph
A column set for graphs that were generated for this FCSAnalysis. Graph columns display nicely on LabKey, but their underlying value is not interesting. They are a lookup where the display field is the name of the graph if the graph exists, or null if the graph does not exist.
FCSFile
The FCSFile that this FCSAnalysis was performed on. This is a lookup to the FCSFiles table.
In addition, the FCSAnalyses table defines the methods Graph, and Statistic.
CompensationControls Table
The CompensationControls table lists the analyses of the FCS files that were used to calculate compensation matrices. Often (as in the case of a universal negative) multiple CompensationControls are created for a single FCS file. The CompensationControls table has the following columns in it:
RowId
A unique identifier for the compensation control
Name
The name of the compensation control. This is the channel that it was used for, followed by either “+”, or “-“
Flag
A flag column for the user to add a comment to this compensation control.
Created
The date that this compensation control was created.
Protocol
(hidden)
Run
The run that this compensation control belongs to. This is the run for the compensation calculation, not the run that the FCS file belongs to.
Statistic
A column set for statistics that were calculated for this compensation control. The following statistics are calculated for a compensation control:
comp:Count
The number of events in the relevant population.
comp:Freq_Of_Parent
The fraction of events that made it through the last gate that was applied in the compensation calculation. This value will be 0 if no gates were applied to the compensation control.
comp:Median(channelName)
The median value of the channelName
Graph
A column set for graphs that were generated for this compensation control. The names of graphs for compensation controls are of the form:
comp(channelName)
or
comp(<channelName>)
The latter is shows the post-compensation graph.
In addition, the CompensationControls table defines the methods Statistic and Graph.
AnalysisScripts Table
The AnalysisScripts table lists the analysis scripts in the folder. This table has the following columns:
RowId
A unique identifier for this analysis script.
Name
The user-assigned name of this analysis script
Flag
A flag column for the user to add a comment to this analysis script.
Created
The date this analysis script was created.
Protocol
(hidden)
Run
(hidden)
Analyses Table
The Analyses table lists the experiments in the folder with the exception of the one named Flow Experiment Runs. This table has the following columns:
RowId
A unique identifier
LSID
(hidden)
Name
Hypothesis
Comments
Created
CreatedBy
Modified
ModifiedBy
Container
CompensationRunCount
The number of compensation calculations in this analysis. It is displayed as a hyperlink to the list of compensation runs.
AnalysisRunCount
The number of runs that have been analyzed in this analysis. It is displayed as a hyperlink to the list of those run analyses
Analysis Archive Format
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
The LabKey flow module supports importing and exporting analyses as a series of .tsv and supporting files in a zip archive. The format is intended to be simple for tools to reformat the results of an external analysis engine for importing into LabKey. Notably, the analysis definition is not included in the archive, but may be defined elsewhere in a FlowJo workspace gating hierarchy, an R flowCore script, or be defined by some other software package.
From the flow Runs or FCSAnalysis grid, you can export the analysis results including the original FCS files, keywords, compensation matrices, and statistics.
Open the analysis and select the runs to export.
Select (Export).
Click the Analysis tab.
Make the selections you need and click Export.
Import an Analysis Archive
To import a flow analysis archive, perhaps after making changes outside the server to add different statistics, graphs, or other information, follow these steps:
In the flow folder, Flow Summary web part, click Upload and Import.
Drag and drop the analysis archive into the upload panel.
Select the archive and click Import Data.
In the popup, confirm that Import External Analysis is selected.
Click Import.
Analysis Archive Format
In brief, the archive format contains the following files:
All analysis tsv files are optional. The keywords.tsv file lists the keywords for each sample. The statistics.tsv file contains summary statistic values for each sample in the analysis grouped by population. The graphs.tsv contains a catalog of graph images for each sample where the image format may be any image format (pdf, png, svg, etc.) The compensation.tsv contains a catalog of compensation matrices. To keep the directory listing clean, the graphs or compensation matrices may be grouped into sub-directories. For example, the graph images for each sample could be placed into a directory with the same name as the sample.
ACS Container Format
The ACS container format is not sufficient for direct import to LabKey. The ACS table of contents only includes relationships between files and doesn’t include, for example, the population name and channel/parameter used to calculate a statistic or render a graph. If the ACS ToC could include those missing metadata, the graphs.tsv would be made redundant. The statistics.tsv would still be needed, however.If you have analyzed results tsv files bundled inside an ACS container, you may be able to extract portions of the files for reformatting into the LabKey flow analysis archive zip format, but you would need to generate the graphs.tsv file manually.
Statistics File
The statistics.tsv file is a tab-separated list of values containing stat names and values. The statistic values may be grouped in a few different ways: (a) no grouping (one statistic value per line), (b) grouped by sample (each column is a new statistic), (c) grouped by sample and population (the current default encoding), or (d) grouped by sample, population, and channel.
Sample Name
Samples are identified by the value in the sample column so must be unique in the analysis. Usually the sample name is just the FCS file name including the ‘.fcs’ extension (e.g., “12345.fcs”).
Population Name
The population column is a unique name within the analysis that identifies the set of events that the statistics were calculated from. A common way to identify the statistics is to use the gating path with gate names separated by a forward slash. If the population name starts with “(” or contains one of “/”, “{”, or “}” the population name must be escaped. To escape illegal characters, wrap the entire gate name in curly brackets { }. For example, the population “A/{B/C}” is the sub-population “B/C” of population “A”.
Statistic Name
The statistic is encoded in the column header as statistic(parameter:percentile) where the parameter and percentile portions are required depending upon the statistic type. The statistic part of the column header may be either the short name (“%P”) or the long name (“Frequency_Of_Parent”). The parameter part is required for the frequency of ancestor statistic and for other channel based statistics. The frequency of ancestor statistic uses the name of an ancestor population as the parameter value while the other statistics use a channel name as the parameter value. To represent compensated parameters, the channel name is wrapped in angle brackets, e.g “<FITC-A>”. The percentile part is required only by the “Percentile” statistic and is an integer in the range of 1-99.The statistic value is a either an integer number or a double. Count stats are integer values >= 0. Percentage stats are doubles in the range 0-100. Other stats are doubles. If the statistic is not present for the given sample and population, it is left blank.
Allowed Statistics
Short Name
Long Name
Parameter
Type
Count
Count
n/a
Integer
%
Frequency
n/a
Double (0-100)
%P
Frequency_Of_Parent
n/a
Double (0-100)
%G
Frequency_Of_Grandparent
n/a
Double (0-100)
%of
Frequency_Of_Ancestor
ancestor population name
Double (0-100)
Min
Min
channel name
Double
Max
Max
channel name
Double
Median
Median
channel name
Double
Mean
Mean
channel name
Double
GeomMean
Geometric_Mean
channel name
Double
StdDev
Std_Dev
channel name
Double
rStdDev
Robust_Std_Dev
channel name
Double
MAD
Median_Abs_Dev
channel name
Double
MAD%
Median_Abs_Dev_Percent
channel name
Double (0-100)
CV
CV
channel name
Double
rCV
Robust_CV
channel name
Double
%ile
Percentile
channel name and percentile 1-99
Double (0-100)
For example, the following are valid statistic names:
Count
Robust_CV(<FITC>)
%ile(<Pacific-Blue>:30)
%of(Lymphocytes)
Examples
NOTE: The following examples are for illustration purposes only.
No Grouping: One Row Per Sample and Statistic
The required columns are Sample, Population, Statistic, and Value. No extra columns are present. Each statistic is on a new line.
Sample
Population
Statistic
Value
Sample1.fcs
S/L/Lv/3+/4+/IFNg+IL2+
%P
0.85
Sample1.fcs
S/L/Lv/3+/4+/IFNg+IL2-
Count
12001
Sample2.fcs
S/L/Lv/3+/{escaped/slash}
Median(FITC-A)
23,000
Sample2.fcs
S/L/Lv/3+/4+/IFNg+IL2+
%ile(<Pacific-Blue>:30)
0.93
Grouped By Sample
The only required column is Sample. The remaining columns are statistic columns where the column name contain the population name and statistic name separated by a colon.
Sample
S/L/Lv/3+/4+/IFNg+IL2+:Count
S/L/Lv/3+/4+/IFNg+IL2+:%P
S/L/Lv/3+/4+/IFNg+IL2-:%ile(<Pacific-Blue>:30)
S/L/Lv/3+/4+/IFNg+IL2-:%P
Sample1.fcs
12001
0.93
12314
0.24
Sample2.fcs
13056
0.85
13023
0.56
Grouped By Sample and Population
The required columns are Sample and Population. The remaining columns are statistic names including any required parameter part and percentile part.
Sample
Population
Count
%P
Median(FITC-A)
%ile(<Pacific-Blue>:30)
Sample1.fcs
S/L/Lv/3+/4+/IFNg+IL2+
12001
0.93
45223
12314
Sample1.fcs
S/L/Lv/3+/4+/IFNg+IL2-
12312
0.94
12345
Sample2.fcs
S/L/Lv/3+/4+/IFNg+IL2+
13056
0.85
13023
Sample2.fcs
S/L/Lv/{slash/escaped}
3042
0.35
13023
Grouped By Sample, Population, and Parameter
The required columns are Sample, Population, and Parameter. The remaining columns are statistic names with any required percentile part.
Sample
Population
Parameter
Count
%P
Median
%ile(30)
Sample1.fcs
S/L/Lv/3+/4+/IFNg+IL2+
12001
0.93
Sample1.fcs
S/L/Lv/3+/4+/IFNg+IL2+
FITC-A
45223
Sample1.fcs
S/L/Lv/3+/4+/IFNg+IL2+
<Pacific-Blue>
12314
Graphs File
The graphs.tsv file is a catalog of plot images generated by the analysis. It is similar to the statistics file and lists the sample name, plot file name, and plot parameters. Currently, the only plot parameters included in the graphs.tsv are the population and x and y axes. The graph.tsv file contains one graph image per row. The population column is encoded in the same manner as in the statistics.tsv file. The graph column is the colon-concatenated x and y axes used to render the plot. Compensated parameters are surrounded with <> angle brackets. (Future formats may split x and y axes into separate columns to ease parsing.) The path is a relative file path to the image (no “.” or “..” is allowed in the path) and the image name is usually just an MD5-sum of the graph bytes.Multi-sample or multi-plot images are not yet supported.
Sample
Population
Graph
Path
Sample1.fcs
S/L/Lv/3+/4+/IFNg+IL2+
<APC-A>
sample01/graph01.png
Sample1.fcs
S/L/Lv/3+/4+/IFNg+IL2-
SSC-A:<APC-A>
sample01/graph02.png
Sample2.fcs
S/L/Lv/3+/4+/IFNg+IL2+
FSC-H:FSC-A
sample02/graph01.svg
...
Compensation File
The compensation.tsv file maps sample names to compensation matrix file paths. The required columns are Sample and Path. The path is a relative file path to the matrix (no “.” or “..” is allowed in the path). The comp. matrix file is in the FlowJo comp matrix file format or a GatingML transforms:spilloverMatrix XML document.
Sample
Path
Sample1.fcs
compensation/matrix1
Sample2.fcs
compensation/matrix2.xml
Keywords File
The keywords.tsv lists the keyword names and values for each sample. This file has the required columns Sample, Keyword, and Value.
Sample
Keyword
Value
Sample1.fcs
$MODE
L
Sample1.fcs
$DATATYPE
F
...
Add Flow Data to a Study
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
For Flow data to be added to a study, it must include participant/timepoint ids or specimen ids, which LabKey Server uses to align the data into a structured, longitudinal study. The topic below describes four available mechanisms for supplying these ids to Flow data.
Add keywords to the flow data before importing them into LabKey. If your flow data already has keywords for either the SpecimenId or for ParticipantId and Timepoints, then you can link the Flow data into a study without further modification.You can add keywords to an FCS file in most acquisition software, such as FlowJo. You can also add keywords in the WSP file, which LabKey will pick up. Use this method when you have control over the Flow source files, and if it is convenient to change them before import.
2. Add Keywords After Import to LabKey
If your flow data does not already contain the appropriate keywords, you can add them after import to LabKey Server. Note this method does not change the original FCS or WSP files. The additional keyword data only resides inside LabKey Server. Use this method when you cannot change the source Flow files, or when it is undesirable to do so.
Navigate to the imported FCS files.
Select one or more files from the grid. The group of files you select should apply to one participant.
Click Edit Keywords.
Scroll down to the bottom of the page and click Create a New Keyword.
Add keywords appropriate for your target study. There are three options:
This method extends the information about the flow samples to include participant ids, visits, etc. It uses a sample type as a mapping table, associating participant/visit metadata with the flow vials.For example, if you had Flow data like the following:
TubeName
PLATE_ID
FlowJoFileID
and so on...
B1
123
4110886493
...
B2
345
3946880114
...
B3
789
8693541319
...
You could extend the fields with a sample type like
TubeName
PTID
Date
Visit
B1
202
2009-01-18
3
B2
202
2008-11-23
2
B3
202
2008-10-04
1
Add a sample type that contains the ptid/visit data.
Associate the sample type with the flow data (FCS files).
Under Assign additional meanings to keywords, click Define/Modify Sample Description Join Fields.
On the left select the vial name in the flow data; on the right select vial name in the sample type. This extends the descriptions of the flow vials to include the fields in the sample type.
To view the extended descriptions: Under Analysis Folders, click the analysis you want to link to a study. Click the run you wish to link. You may need to use (Grid Views) > Customize Grid to add the ptid/visit fields.
To link to study: select from the grid and click Link to Study.
4. Add Participant/Visit Data During the Link-to-Study Process
You can manually add participant/visit data as part of the link-to-study wizard. For details see Link Assay Data into a Study.
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
The keywords.jar file attached to this page is a simple commandline tool to dump the keywords from a set of FCS files. Used together with findstr or grep this can be used to search a directory of fcs files.Download the jar file: keywords.jarThe following will show you all the 'interesting' keywords from all the files in the current directory (most of the $ keywords are hidden).
java -jar keywords.jar *.fcs
The following will show the EXPERIMENT ID, Stim, and $Tot keywords for each fcs file. You may need to escape the '$' on linux command line shells.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
The FluoroSpot assay is a variant of the ELISpot assays, but in place of enzymes, FluoroSpot uses fluorophores to signal cell secretion. The use of variously colored fluorophores in a single well lets a researcher detect several secreted analytes simultaneously.
The FluoroSpot assay type built in to LabKey Server assumes that your data has been output as multi-sheet Excel files from an AID MultiSpot reader.
FluoroSpot Assay Set Up
The follow instructions explain how to create a new FluoroSpot plate template and assay design (based on the FluoroSpot assay type).
Navigate to, or create, a folder for your fluorospot assay tutorial. You can use the Assay Tutorial folder if you created it for another tutorial.
Select > Manage Assays.
Click Configure Plate Templates.
Select new 96 well (8x12) ELISpot default template from the dropdown.
Click Create to open the editor.
Name and configure the plate as fits your experiment. In this example, we've named the plate "FluoroSpot Template". When finished, click Save and Close.
Select > Manage Assays.
Click New Assay Design.
Click the Specialty Assays tab.
Select ELISpot and your current folder as the Assay Location, then click Choose ELISpot Assay.
Name your new assay design.
In the Assay Properties section, on the Plate Template dropdown, select the plate template you just defined.
In the Assay Properties section, on the Detection Methods dropdown, select 'fluorescent'.
Set the Batch, Run, Sample, Antigen, and Analyte fields as appropriate to your experiment.
Scroll down and click Save.
Importing Data to the FluoroSpot Design
To import data into the assay design, you can use the "Import Data" button, as shown below; or, as an alternative, you can use the Files web part (for details, see Import Data from Files). Note that data import for the FluoroSpot assay is similar to data import for the ELISpot assay -- to compare these processes see: Tutorial: ELISpot Assay Tutorial.
On the Assay List, select your FluoroSpot assay design.
Click Import Data
Enter any Batch Properties for your experiment.
If you plan to integrate the assay data into a study, specify how the data maps to other data you have collected. For details see Participant/Visit Resolver Field. The actual data entry for participants, visits, and/or specimens takes place in the next step.
Click Next.
Enter any Run Properties for your experiment, for example, you can specify the lab which performed the assay runs.
Select the Plate Reader(Required). For our example files, select "AID".
Select Choose File and select the spreadsheet of data to be imported (either of our example files).
Enter the ParticipantId, VisitId, or SpecimenId, depending on how this information is encoded.
Click Next.
Enter the Antigen Properties for your experiment, including the Antigen name(s), id(s) and the number of cells per well.
Click Next.
On the Analyte Properties page, enter the cytokine names used in the experiment.
Click Save and Finish, or Save and Import Another Run if you have more spreadsheets to enter.
FluoroSpot Statistics and Plate Summary
Once data import is complete, you will be taken to the "Runs" view of your data.
Hover over the row of interest and click the (details link) in the left column to view statistics (mean and median) grouped by antigen.
Scroll down to see that the "Run Details" view also displays plate summary information for each analyte. Select the sample and antigen to highlight the applicable wells. Hover over a well to see more details.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
LabKey Server's tool for Luminex® assays help you to manage, quality control, analyze, share, integrate and export results from BioPlex instruments. Luminex immunoassays are plate-based assays that can measure multiple analytes independently in each well.Additional information can be found in this paper:
There are two tutorials which introduce the Luminex features using some example data. They are independent of each other, but we recommend completing them in order to learn your way around the tools. The tutorial scenario is that we want to evaluate binding between a panel of HIV envelope proteins (antigens) and plasma immunoglobulin A (antibodies) from participants in a study.
Troubleshooting note: If you see a message like this "Could not find AssayProvider for assay design 'Luminex Assay 200' (id 335)" (substituting your own assay design name and id number), your server may be missing the "luminex" module. Contact your Account Manager to resolve this issue.
Background
LabKey Server supports multiplexed immunoassays based on Luminex xMAP® technology. A Luminex assay multiplexes analysis of up to 500 analytes in each plate well. In contrast, an ELISA requires a separate well for analysis of each individual analyte.Luminex immunoassays often aim to do one or both of the following:
Measure the strength of binding between a set of analytes (e.g., virus envelope antigens) and unknowns (e.g., blood serum samples with unknown antibody activity).
Find concentrations of unknowns using dose-response curves calculated for titrated standards.
LabKey Server supports using an R transform script to customize Luminex analyses and using Levey-Jennings plots for performing cross-run quality control.
Binding and Concentrations
Each analyte is bound to a different type of bead. Each bead contains a mixture of red and infrared dyes in a ratio whose spectral signature can identify the bead (and thus the analyte).In each plate well, analyte-bound beads of many types are combined with a sample. The sample added to each plate well is typically a replicate for one of the following:
A titrated standard whose concentrations are known and used to calculate a reference curve
A titrated quality control used for verification
An unknown, such as serum from a study participant
A background well, which contains no active compound and is used for subtracting background fluorescence
Bead-analyte-sample complexes are rinsed, then allowed to react with a phycoerythrin-bound detection reagent, then rinsed again. The fluorochrome attached to the detection reagent serves as the reporter of binding.Next, each bead is illuminated by two lasers to detect the bead/analyte type (from the red/infrared ratio) and sample binding (from the fluorochrome's fluorescence). The instrument reports the median fluorescence intensity (FI) for all beads attached to a given analyte type for each well, among other measures.
The LabKey Luminex transform script
The LabKey Luminex tool can be configured to run a custom R transform script that applies custom curve fits to titrated standards. The script also uses these curve fits to find estimated concentrations of samples and other desired parameters.The data used by the script for such analyses can be customized based on the latest lab techniques; for example, the default script allows users to make choices in the user interface that enable subtraction of negative bead FI to account for nonspecific binding to beads.The methods and parameters used for the curve fit can be fully customized, including the weighting used for squared errors to account for trends in variance, the equation used to seed the fitting process, and the optimization technique.Developers can customize the R transform script to use the latest R packages to provide results that reflect the most advanced algorithms and statistical techniques. Additional calculations can be provided automatically by the script, such as calculations of a result’s “positivity” (increase relative to a baseline measurement input by the user).
Levey-Jennings plots
Levey-Jennings plots can help labs execute cross-run quality control by visualizing trends and identifying outliers.LabKey Server automatically generates Levey-Jennings plots for a variety of metrics for the curve fits determined from titrations of standards. See Step 5: Track Analyte Quality Over Time for more information.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
LabKey Server tools for Luminex® assays help you to manage, quality control, analyze, share, integrate and export Luminex results.This tutorial is the first of two for Luminex assays, and covers basic procedures specific to working with the BioPlex data. If you are unfamiliar with the general process of creating an assay design and importing data files into the LabKey assay framework, you may find it helpful to first review the material in Tutorial: Import Experimental / Assay Data.In this tutorial you will:
Create a Luminex specific assay design.
Import Luminex assay data and collect additional, pre-defined analyte, run and batch properties for each run or batch of runs.
Exclude an analyte's results from assay results, either for a single replicate group of wells or for all wells.
Import single runs composed of multiple files and associate them as a single run of standards, quality controls and unknowns.
Copy quality-controlled, sample-associated assay results into a LabKey study to allow integration with other types of data.
The Luminex assay runs you import in this tutorial can be seen in the interactive example. This example allows visitors to interact with Luminex features that do not require editor-level or higher permissions.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
The two Luminex tutorials begin with uploading example data into a folder on LabKey Server. You can share the same folder for both tutorials, so only need to complete this page once.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
An assay design is a structured data container designed to collect and store information about a particular assay experiment. Some Specialty Assay types, including Luminex®, provide starting templates to make it simpler to create the design required for a particular instrument. In this tutorial, we simply create a named instance of the default Luminex assay design. You could further customize this assay design if needed, but this tutorial does not do so. For further information on the fields included in the default Luminex design, see Luminex Properties.
Create a New Default Luminex Design
Select > Manage Assays.
Click New Assay Design.
Click the Specialty Assays tab.
Under Use Instrument Specific Data Format, select Luminex.
For the Assay Location choose Current Folder (Luminex).
Click Choose Luminex Assay.
The Luminex Assay Designer page lets you define assay properties and make any changes necessary to the schema. In this example, we use the default Luminex properties.
In the Assay Properties section, for Name enter "Luminex Assay 100".
Review the properties and fields available, but do not change them.
Click Save.
You've now created a new named assay design and can begin importing data.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
In this tutorial step, we import some sample Luminex® run data using the new assay design you just created. You will enter batch, run and analyte properties for a single run contained in a Excel single file. (Other topics describe how to import a multi-file run, how to import a batch of runs at once, and how to reimport a run.)To learn more about the properties and default settings you will accept here, review the topic: Luminex Properties.
Start the Import Process
Return to the main page of the Luminex folder (by clicking the Luminex link near the top of the page).
In the Files web part, open the Luminex folder, and then the Runs - Assay 100 folder.
Select the file 02-14A22-IgA-Biotin.xls by single-clicking it.
Click Import Data.
In the Import Data popup window, select Use Luminex Assay 100 (the design you just created).
Click Import.
Batch Properties
First, we enter batch properties. These are properties which are set once per batch of runs, which in this first example is only a single run.
Participant/Visit and Target Study: Leave the default selections ("Sample information in the data file" and "None". These fields are used for aligning assay data with existing participant and sample data. We will learn more about these fields later in this tutorial.)
Species: Enter "Human".
Lab ID: Enter "LabKey".
Analysis software: Enter "BioPlex"
Click Next.
Run Properties
Next, we can enter properties specific to this run.
On the page Data Import: Run Properties and Data File, leave all fields unchanged.
Note that the Run data field points to the Excel file we are about to import: "02-14A22-IgA-Biotin.xls". When you leave the Assay Id field blank (as you do in this step), the full name of the imported file will be used as the Assay Id.
Click Next.
Analyte Properties
While any assay may have batch or run properties, some properties are particular to the specific type of assay. Analyte properties and Well Roles defined on this page are an example for Luminex assays. Learn more in this topic: Luminex Properties.
Well Roles
If a sample appears at several different dilutions, we infer you have titrated it. During the import process for the run, you can indicate whether you are using the titrated sample as a standard, quality control, or unknown. Here, we elect to use Standard1 (the standard in the imported file) as a Standard.In this panel you will also see checkboxes for Tracked Single Point Controls. Check a box if you would like to generate a Levey-Jennings report to track the performance of a single-point control. Learn more in the Level II tutorial which includes: Step 5: Track Analyte Quality Over Time and Track Single-Point Controls in Levey-Jennings Plots.
Analyte Properties
The Analyte Properties section is used to supply properties that may be specific to each analyte in the run, or shared by all analytes in the run. For example, you could track bead or analyte lots by setting properties here.
On the page Data Import: Analyte Properties leave the sections Define Well Roles and Analyte Properties unchanged.
Click Save and Finish.
View Results
The data has been imported, and you can view the results.
In the Luminex Assay 100 Runs table, click 02-14A22-IgA-Biotin.xls to see the imported data for the run.
The data grid will look something like this:
You can see a similar page in the interactive example. (Note that there have been exclusions applied to the example following a later step in this tutorial.)
Note that some views of Luminex assay data will have filters already applied. These are listed in the Filter: panel above the data grid. Filters can be cleared by clicking the for each filter item.
After excluding some analytes in the next step, you will reimport this run to see that when reimporting, the properties you entered are retained simplifying subsequent imports.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
In some cases you may want to flag certain wells and/or analytes as unreliable, in order to exclude the unreliable data from later analysis.A replicate group is a set of wells that all contain the same sample at the same dilution. For example, the replicate groups in the sample data file used in this tutorial each encompass two wells. Each pair of wells contains the same unknown sample at the same dilution. In the data file, wells that are part of a replicate group are listed in sequential rows and have the same value in the "Type" column; for example, the two wells with a "Type" of "X1" are part of one replicate group.For example, you might wish to exclude an analyte's results for all sample wells in a replicate group because fluorescence intensities reported for that analyte for that group had high coefficients of variation. Alternatively, you might wish to exclude all measurements for the entire run for a particular analyte after discovering that this analyte was bound to beads from a defective lot.
Exclusion Options: There are several exclusion options in LabKey Server's Luminex® Assay tools.
Note that when data is excluded, the assay run including transform script curve fits will be recalculated without that data which can take considerable time. These exclusion reruns are pushed onto the pipeline job list so that you may continue working and check on status later. When you define a new exclusion you may opt to view the status log immediately in order to wait for completion instead.
Exclude Analytes for a Replicate Group
Here, you will exclude a single analyte for all sample wells within a replicate group, in this case there are two wells per group.In this example scenario, since the fluorescence intensity reported for an analyte in a well is the median for all beads bound to that analyte in that well, you might exclude the analyte if the reported coefficient of variation for the group was unusually high.
Filter Results to a Replicate Group
Filter the results grid so that you see only unknowns, not background, standard or control wells.
On the Luminex Assay 100 Results page, click the column header for the Well Role column.
Select Filter.
In the filter popup on the Choose Values tab, select the Unknown option only. Click the label Unknown to select only that checkbox.
Click OK.
Notice that the grid now shows your new filter, as well as a filter to show only the specific run. Before continuing to the next step of modifying the default grid view, clear the run filter by clicking the 'X' (your run number will likely be different). Otherwise the default grid view will only show this particular run going forward.Customize the results grid to show instrument-reported coefficients of variation for each well:
Select (Grid Views) > Customize Grid.
In the Available Fields panel, scroll down, click the box for CV, then click Save. Confirm that "Default grid view for this page" is selected, and click Save again.
Scroll to the far right and note that the first two wells for ENV1 (a replicate pair) show much higher CV than the other wells for this analyte.
Scroll back to the left.
Exclude Data for the Replicate Group
To exclude data for a replicate group, click the (circle/slash) icon for a row in the group. Select the replicate group using the checkbox, then choose whether all analytes for the sample group are excluded, or just the row you picked initially.
Click the (circle/slash) icon on the second row:
In the Exclude Well or Replicate Group from Analysis popup:
Check the box for "Replicate Group".
Click Exclude selected analytes.
Check the box for ENV1 to exclude this analyte.
This exclusion will apply to analyte ENV1 in a replicate group that includes wells A1 and B1.
Click Save.
A popup message will explain that the exclusion will run in the background, via the pipeline - you have the option to view the pipeline status log to await completion. For this tutorial, just click No and continue; this small exclusion will run quickly.
To undo a replicate group exclusion, you would again click on the (circle/slash) icon for the excluded row, then uncheck the relevant analytes in the popup and click Save again.
Exclusion Color Coding and Reasons
Once excluded, the rows are color coded with red/pink and the Flagged As Excluded field changes value from "no" to "yes". Use this field to sort and filter the assay data, if desired.
Refresh the Luminex Assay 100 Results page by clicking View Results.
For additional detail about why a given row is marked as excluded, you can customize the grid view to include the Exclusion Comment field. This can help you identify whether the row is excluded as part of a well group, analyte exclusion, etc.
Exclude Analytes Regardless of Replicate Group
Next, we exclude a given analyte for all wells in the run. For example, you might wish to exclude all measurements for a particular analyte if you realized that this analyte was bound to beads that came from a problematic lot.
On the Luminex Assay 100 Results grid, select Exclusions > Exclude Analytes.
Where is the Exclusions menu option? You must have editor or higher permissions to see this option. If you still don't see the option, your results view is probably unfiltered. The button only appears for results that have been filtered to one particular run.
To make the button appear, click Luminex Assay 100 Runs at the top of the page, then click the run name -- this filtered result view will include the Exclusions menu if you have editor or higher permissions.
In the Exclude Analytes from Analysis popup, select ENV5.
Notice that there is no Wells field; exclusions apply to all replicate groups.
Click Save.
Click No in the popup message offering the option to view the pipeline status.
To remove an analyte exclusion, select Exclusions > Exclude Analytes again. In the popup dialog, clear any exclusion checkmarks, and click Save.
Exclude Titration
When you exclude data from a well-group within a titration, the assay is re-calculated (i.e. the transform script is rerun and data is replaced). If you want to exclude all well groups for a given titration, you can do so without recalculating by selecting Exclusions > Exclude Titration and specifying which titration to exclude. Note that you cannot exclude a "Standard" titration.
The sample data used for this tutorial does not include an excludable titration. If it did, there would be an Exclude Titration option under Exclude Analytes on the Exclusions menu.
The exclusion popup in this case lets you select each titration present and check which analytes to exclude for that titration. Note that analytes excluded already for a replicate group, singlepoint unknown, or at the assay level will not be re-included by changes in titration exclusion.
Exclude Singlepoint Unknowns
To exclude analytes from singlepoint unknown samples, select Exclusions > Exclude Singlepoint Unknowns. The unknown samples are listed, and you can select each one in turn and choose one or more analytes to exclude from only that singlepoint.Note that analytes excluded for a replicate group, titration, or at the assay level will not be re-included by changes in singlepoint unknown exclusions.
Exclude Individual Wells
When you open the UI to exclude a replicate group, the default option is to only exclude the single well for the row you selected. You can choose to exclude all analytes or only selected analytes for that well.
View All Excluded Data
Click View Excluded Data above the results grid to see all exclusions in a single page. This might help you see which data needs to be re-run. You could of course view excluded rows by filtering the grid view on the Flagged as Excluded column, but the summary page gives a summary of exclusions across multiple runs.If you are looking at the results grid for a single run, the View Excluded Data report will be filtered to show the exclusions for that run.
Return to the list of runs by clicking View Runs. If you wanted to reimport the run, perhaps to recalculate positivity, you could do so by selecting the run and clicking Reimport Run. Note that the previous run data will be deleted and overwritten with the new run data.You will see the same entry screens for batch and run properties as for a new run, but the values you entered previously will now appear as defaults. You can change them for the rerun as necessary. If exclusions have been applied to the run being replaced, the analyte properties page will include an Exclusion Warning panel.The panel lists how many exclusions are applied (in this case one replicate group, one analyte, and one single point control). Click Exclusions Report to review the same report as you saw using the "View Excluded Data" link from the run data grid. Check the Retain matched exclusions box to retain them; then click Save and Finish to initiate the reimport.If any exclusions would no longer match based on the reimport run data, you will see a red warning message saying the exclusion will be lost if you proceed with the import.[ Video Overview: Retain Luminex Exclusions on Reimport ]When the reimport is complete, click the name of the run to see the reimported data, with any exclusions you chose to retain.
If you don't see data at this point, you may have saved the default view of your grid with a run filter. Now that the run has been reimported, the run ID is different. To resolve this issue:
Select (Grid Views) > Customize Grid.
Click the Filters tab.
Clear the Row ID filter by clicking the 'X' on it's line.
Click Save, confirm Default grid view for this page is selected.
Click Save again.
Edit or Remove Exclusions
To edit an exclusion, reopen the panel where you originally added the exclusion at that level. Adjust the settings and save. As with the original exclusion, this operation will run in the background.To remove an exclusion, edit it and remove all selections before saving.
Does LabKey Server re-calculate titration curves when I exclude a replicate group?
This happens only when your assay design is associated with a transform script (see: Luminex Assay Tutorial Level II). When you exclude an entire replicate group from a titrated standard or quality control, LabKey Server automatically re-runs the transform script for that run.This is desirable because changing the replicates included in a titration affects the calculations that the script performs for the curve fits and other measure (e.g., EC50, AUC, HighMFI, etc., as defined in Luminex Calculations).If you exclude a replicate group that is not part of a titration (e.g., part of an unknown), the calculations performed by the script will be unaffected, so the script is not re-run.
Does LabKey Server do automatic flagging of run data outliers during data import?
During data import, LabKey Server adds a quality control flag to each data row where reported FI is greater than 100 and%CV (coefficient of variation) is outside of a certain threshold. For unknowns, the %CV must be greater than 15 to receive a flag; for standards and quality controls, the %CV must be greater than 20.Columns flagged through this process are highlighted in red. If flagging is later disabled, the row is no longer highlighted.To see all rows that received flags according to these threshholds, you can add the CVQCFlagsEnabled column to a Luminex data or results view and filter the data using this column. The column is hidden by default.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
This step describes how to import several files of Luminex® results as part of one run. Note that this is not the same as importing several single-file runs as one batch.A single run may span several files for different reasons:
You have too many unknowns to fit on one plate, so you place them on several different plates that you run together. Since each plate produces an Excel file, you have several Excel files for that run.
You have one standard plate that you need to associate with several different runs of unknowns. In this case, you run the standards only once on their own plate and associate them with several runs of unknowns. You might do this if you had only a limited supply of an expensive standard.
This example follows the second circumstance. Our standards and unknowns are in separate files, but they should be considered part of the same run.If you do not need to work with multi-file runs, you can skip this step and proceed to Step 5: Link Luminex Data to Study.
Import Multi-File Runs
In this example, we want to import two files as a single run. The two files are:
File A (01-11A12a-IgA-Biotin.xls) reports results for the standard.
File B (01-11A12b-IgA-Biotin.xls) reports results for unknowns that are associated with this standard.
Select > Manage Assays.
In the Assay List web part, click the Luminex Assay 100 assay design.
Click Import Data.
Leave the batch property default values in place, and click Next.
In Run Properties:
Scroll down to the Run Data property.
Click the Choose Files button and navigate to the LuminexExample files you downloaded and unzipped earlier.
In the directory Runs - Assay 100/MultiFile Runs/ you can multi-select the two files: 01-11A12a-IgA-Biotin.xls and 01-11A12b-IgA-Biotin.xls and click Open.
You could also select one file at a time, using the button to add the second file.
Click Next.
On the Data Import: Analyte Properties page, leave all default values in place.
If any cross-plate titrations were found in your upload, you will be asked to confirm whether they should be inferred as titrated unknowns here.
Click Save And Finish.
The new run, 01-11A12a-IgA-Biotin.xls, appears in the Luminex Assay 100 Runs list. Since we did not provide an Assay ID for this run during the import process, the ID is the name of the first file imported.
The new multi-file run appears in the grid alongside the single-file run you imported earlier. Reviewing data and excluding analytes work the same way for both types of run.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Linking assay data into a study allows you to integrate it with other types of data from other sources. If you wish, you can skip this step and proceed to the next Luminex tutorial: Luminex Assay Tutorial Level II.In the steps below, we move selected, quality-controlled Luminex assay data into a LabKey study. In this example, we select only data associated with unknowns, and only data that have not been excluded as part of the QC process.In order to integrate assay data with other types of data in a LabKey study, you need to connect the instrument data to participants or specimen samples in some way. This step shows you how to link the sample identifiers in your instrument data with specimen vial identifiers in the target study. The links are provided in a mapping file that associates the sample identifiers (in the Description column, e.g., "111", "112", "113") with participant and visit identifiers in our target study. If you do not provide a mapping file, you would need to manually enter the participant identifier and visit for each row of data.
Install a Target Study
You need a target study for your assay data, so if you don't already have one, create one here:
To install the target study, complete the instructions in this topic: Install the Demo Study
ReImport a Run
Next we re-import a run, this time indicating a target study and a mapping file for the data.
Return to your Luminex folder.
In the Assay List, click Luminex Assay 100.
On the Luminex Assay 100 Runs list, place a checkmark next to 02-14A22-IgA-Biotin.xls (the original single run we imported).
Click Re-Import Run.
Under Batch Properties enter the following:
Participant/Visit:
Click the checkbox Sample indices, which map to values in a different data source.
In the LuminexExample data you downloaded and unzipped, open the Excel file /Luminex/Runs - Assay 100/IndexMap.xls.
Copy and paste the entire contents of this file (including all column headers and non-blank rows) to the text box under Paste a sample list as a TSV.
Target Study:
Select the target study you installed above.
Click Next.
On the page Data Import: Run Properties and Data File, scroll down and click Next.
On the page Data Import: Analyte Properties you may see a warning about previous exclusions; leave the "Retain matched exclusions" checkbox checked if so.
Scroll down and click Save and Finish.
Select a Subset of Data to Link
We want to exclude standards, controls and background wells in the data we link. We also plan to exclude data that was flagged and excluded as part of the QC process.
On the page Luminex Assay 100 Runs click 02-14A22-IgA-Biotin.xls (reimporting made it the top row).
Notice that your data grid now includes values in the Specimen ID, Participant ID, and visitID columns which were not present during the earlier tutorial steps. We populated them by pasting the IndexMap values during the reimport.
Filter the results to show only values we were able to map to study participants.
Click the Participant ID column header and select Filter.
In the popup, click Choose Filters, then the filter type Is Not Blank, then click OK.
Filter the results to show only wells with non-excluded data:
Click the Flagged As Excluded column header and select Filter.
In the popup, select only false and click OK.
If it is not already filtered by your saved grid view, filter the results to show only wells with unknowns (vs standards or controls):
Click the Well Role column header and select Filter.
If "Unknown" is the only option, you don't need to take any action. If not, select Unknown by clicking its label, and click OK.
Select all displayed rows on the current page of the grid using the checkbox at the top of the left column.
You will see a message reading "Selected 100 of ### rows" alongside other select/show options. Notes: the total number of rows may vary based on the exclusions and filters you set. Also, '100' is the default grid paging option but other settings are available.
Click Select All Rows.
Link Selected Data to the Study
Click Link to Study.
Note that we have pre-selected our target study and you will confirm that it is the target. If you wanted, you could choose a different target study, and/or specify a dataset Category for the linked data.
Click Next.
The data in this case will be successfully matched to participants/visits in our example study. You will see (checkmarks) next to each row of data that has been successfully matched, as shown in the screen shot below.
Finalize by clicking Link To Study.
If you see the error message 'You must specify a Participant ID (or Visit/Date) for all rows.' it means there are rows without this data. Scroll down to uncheck the rows for them, or provide these details, then click Re-validate to proceed with the link.
When the linkage is complete, you will be in the target study viewing the linked dataset. If other data from Luminex Assay 100 has already been linked to the study, the new dataset may be shown as Luminex Assay 101 or 1001 or something similar.
View the Linked Data in the Study
To see the dataset in the study, click the Clinical and Assay Data tab.
Find your linked Luminex dataset on the list, and click to view the data.
Click View Source Assay to return to the original Luminex assay folder.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
LabKey Server's tool for Luminex® assays can help you to manage, quality control, analyze, share, integrate and export Luminex results.This tutorial builds on an understanding of material covered in Luminex Assay Tutorial Level I and shows you how to:
Import Luminex assay data stored in structured Excel files that have been output from a BioPlex instrument.
Set values during import of pre-defined analyte, run and batch properties for each analyte, run and/or batch of runs.
Run the LabKey Luminex transform script during import to calculate logistic curve fits and other parameters for standard titrations.
View curve fits and calculated values for each standard titration.
Visualize changes in the performance of standards over time using Levey-Jennings plots.
Determine expected ranges for performance of standards for analyte lots, then flag exceptional values.
Note that for simplicity, the example datasets used in this tutorial include only standard titrations. All steps covered here could equally apply to quality control titrations.Further detail and background on many of the steps in this tutorial can be found in the Luminex Reference documentationThe Luminex assay data you will import in this tutorial can also be seen in the interactive example. There you can explore features that do not require editor-level or higher permissions.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
In this step, we import two pre-prepared archives to simplify creating a more complex assay design than we used in the first tutorial. These define:
A set of lists used by the assay design to define lookups for several properties
A pre-prepared assay design for a Luminex® experiment
Set Up
If you have not already set up the Luminex tutorial project, follow this topic to do so: Set Up Luminex Tutorial Folder.
Return to this page when you have completed the set up.
Import the List Archive
The lists imported here define sets of acceptable values for various properties included in the assay design that you will import in a later step. These acceptable values are used to provide drop-down lists to users importing runs to simplify data entry.
Import the List Archive
Navigate to your Luminex tutorial folder.
Select > Manage Lists.
Click Import List Archive.
Click Browse/Choose File and from the Luminex example data you downloaded, select Luminex_ListArchive.lists.zip.
Click Import List Archive.
Import the Assay Design Archive
Next, you will import a pre-prepared assay design which will be used to capture Luminex data.
Click the Luminex link to return to the main folder page.
In the Files web part, double click the Luminex folder.
Select Luminex Assay 200.xar.
Click Import Data.
In the popup dialog, confirm that Import Experiment is selected and click Import.
Refresh your browser, and in the Assay List, the new assay design "Luminex Assay 200" will appear.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Luminex® analysis in LabKey Server makes use of a transform script to do a number of calculations including curve fits and estimated concentrations. To get the transform script running, you will need to:
You will need to install R and configure it as a scripting language on your LabKey Server. If you are on a Windows machine, install R in a directory that does not contain a space (i.e. not the default "C:\Program Files\R" location).
The instructions in this section describe package installation using the R graphical user interface.
Open your R console.
Using the R console, install the packages listed below using commands like the following (you may want to vary the value of repos depending on your geographic location):
Next, place the transform script and utility script in a server-accessible, protected location. For example, if your server runs on Windows you could follow these steps:
Locate the <LK_ENLISTMENT> directory on your local machine. For example, it might be C:\labkey\labkeyEnlistment
Create a new directory named scripts here.
Place a copy of each of the files you downloaded in this new directory:
Associate the Transform Script with the Assay Design
Add Path to Assay Design
Edit the transform script path in the assay design we provided to point to this location.
In the Assay List of your Luminex folder, click Luminex Assay 200.
Click Manage Assay Design > Edit assay design.
In the Assay Properties section, for the Transform Script, enter the full path to the scripts you just placed.
Click Save
When you save, the server will verify the script location.
Test Package Configuration
This step is optional, but will confirm that your R script will run as needed in the following tutorial steps.
Click Luminex.
In the Files web part, select /Luminex/Runs - Assay 200/02-14A22-IgA-Biotin.xls.
Click Import Data.
Select Use Luminex Assay 200 in the popup and click Import again.
For Batch properties, leave defaults unchanged and click Next.
For Run Properties, leave defaults unchanged and click Next.
Click Save and Finish.
If there is a problem with the path, or with installed packages or the version of R, error messages will help you figure out what else you need to do (e.g., installing an additional R package or upgrading your version of R). After installing a missing package, you can refresh your browser window to see if additional errors are generated.If your script cannot find youtil.R, make sure it is located in the same directory as the LabKey Luminex transform script. The following should be peers:
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Here we import 10 Luminex runs to have enough data to provide interesting demonstrations of the tools. You could import each individually as we did in the Luminex Level I tutorial, but using a batch import streamlines the process somewhat.
In the Files web part, open the Runs - Assay 200 folder.
Check the boxes for all 10 run files (the run files begin with a two digit number and end with an .xls extension. Ignore the artifacts with other extensions created when you did a test import.)
Click Import Data.
Select the Luminex Assay 200 assay design.
Click Import.
For Batch Properties, leave default values as provided.
Click Next.
In Run Properties, enter the run number (the first two digits of the file name which is listed in the Run Data field) as the Notebook No.
Click Next.
In Define Well Roles, check all the boxes for this tutorial. For information about these options, see Review Well Roles.
In Analyte Properties, edit the Lot Number:
Check the Lot Number/Same box to give the same lot number to all analytes.
Enter "Lot 437" for all analytes when importing runs #01-05.
Enter "Lot 815" for all analytes when importing runs #06-10.
Because the previously entered value will be retained for each import, you only need to edit this field when importing the first and 6th runs.
In Analyte Properties, check the box in the Negative Control column for Blank.
Then check the Subtract Negative Bead/Same box, and select Blank as the value for all other analytes. These properties are explained here.
Click Save And Import Another Run. Wait for the page to refresh.
You will return to the Run Properties page showing an "Upload successful" message and with the next file name in the Run Data field. Note that the previously entered Notebook No value is retained; remember to edit it before clicking Next, then Save and Import Another Run.
Continue this loop entering the new Notebook No for each run and changing the Analyte Lot message at the 6th run to "Lot 815".
Note: a log file is written for each upload. If you upload two files in the same minute, you will be warned that there is already a log file with that name. Simply wait and click Save and Import Another Run again until the upload completes.
Click Save And Finish when finishing the 10th run.
View the Imported Runs and Data
When you are finished importing the batch of 10 runs, you'll see the fully populated runs list. It will look something like this.Viewing DataTo see the results for a single run, click one of the links in the Assay ID column on the runs page. For example, if you click 10-13A12-IgA-Biotin.xls, you will see the data just for the tenth (latest) run, as shown here.Above the grid are links for View Batches, View Runs, and View Results.
View Batches: The ten runs we just imported were all in one batch; you might have uploaded other runs or batches as well.
View Runs: Shows the list of all runs you have imported in all batches. This is the view you saw when import was complete.
View Results: Shows the assay data, or results, for all imported runs in one grid. The results table for all runs has nearly 5,000 rows, as shown here.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
This step introduces some of the values calculated by the server and transform script for each standard's titration, including the 4-parameter logistic curve fits. For each run, the script outputs a PDF that includes plots for curve fits for each analyte. Each plot shows the dose response curve for fluorescence intensity with increasing concentration or reduced dilution. These plots can be useful for examining how well the curves fit the data.For additional background and details about these and other calculations that are performed, see Luminex Calculations.
View Curve Fits
As an example, here we view one of the 4pl curves generated for the tenth run.
In the Luminex folder, select > Manage Assays.
Click Luminex Assay 200.
Click the curve icon in the Curves column for the tenth run (Assay ID 10-13A12-IgA-Biotin.xls).
It will either download a PDF file directly when you click, or if you have multiple options to select from, such as from different import calculations, you'll see a dropdown menu. If you do, select 10-13A12-IgA-Biotin.Standard1_4PL.pdf.
Open the file. Depending on your browser settings, it may open directly or download for you to click to open.
You will see a series of curves like the one below:
You can also see the example by downloading this PDF containing the full set of curves.
Note: The PDF files for these curves for each run were deposited by the LabKey Luminex transform script in the Runs - Assay 200 folder when the script ran during run import.
View Calculated Values
Some calculated values are stored in the results grid with other Luminex data, others are part of the titration qc reports for standards and other titrations. For more information about the calculations, see Luminex Calculations.
View Calculated Values in Titration QC Reports
For the same tenth run, view calculated values.
Return to the Luminex Assay 200 Runs grid. (If it is not still shown in your browser, select > Manage Assays and click Luminex Assay 200.
In the runs list, click the Assay ID "10-13A12-IgA-Biotin.xls."
Click View QC Report > view titration qc report.
The report shows one row for each analyte in this run. You can see a similar one in the interactive example.
Scroll to the right to see columns calculated by the script:
High MFI
Trapezoidal Curve Fit AUC
Since you selected the report for a single run, you will see 6 rows for just that run. To see these values for all runs, scroll back to the left and clear the run filter by clicking the .
View Calculated Values in Results Grid
Click View Runs.
In the runs list, click 10-13A12-IgA-Biotin.xls
In this Results view, scroll to the right to see columns calculated by the script:
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
In this step, we will visualize a trend in the performance of a standard using a Levey-Jennings plot. We will investigate this trend further in the next step, when we add expected ranges (guide sets) to the plots.
Levey-Jennings plots are quality control tools that help you visualize the performance of laboratory standards and quality controls over time, identifying trends and outlying data points. This can help you take corrective measures to ensure that your standards remain reliable yardsticks for your experimental data. See also: Wikipedia article on Laboratory Quality Control.Example usage scenarios for Levey-Jennings plots:
If you see an outlier data point for a standard, you may investigate whether conditions were unusual on the day the data was collected (e.g., building air conditioning was not working). If the standard was not reliable on that day, other data may also be unreliable.
If you see a trend in the standard (as we will observe below), you may investigate whether experimental conditions are changing (e.g., a reagent is gradually degrading).
If standard performance changes with analyte lot, you may need to investigate the quality of the new analyte lot and potentially change the preparation or supplier of the lot.
The LabKey Luminex tool makes a set of Levey-Jennings plots available for standards for each trio of analyte, isotype and conjugate provided in the run data. Each set of plots for standards includes tabs for three different performance metrics (EC50 4PL, AUC and HighMFI). You can also generate Levey-Jennings plots for single point controls to track performance over time of controls which are not titrated.To see which reports are available for your assay:
Which reports you see here depends on which well role boxes were checked during import. See Review Well Roles for additional information.
Click a link to open a report.
Explore Levey-Jennings Plots for a Standard
The tutorial example includes only a single titration, so we will elect to display Levey-Jennings plots and data for the standard for the ENV2 analyte, IgA isotype and Biotin conjugate trio.
In the list of available reports, click Standard1.
In the Choose Graph Parameters box on the left side, select ENV2.
For Isotype, choose "IgA"
For Conjugate, choose "Biotin".
Click Apply.
Note: at this point in the tutorial, it is possible that you will need to add additional packages to your installation of R to support these plots. Refer to the list in Step 2: Configure R, Packages and Script, or add packages as they are requested by error messages in the UI. Retry viewing the plot after each addition until successful.
In the graph panel, you see a Levey-Jennings plot of EC50 - 4PL for the standard (Standard1).
Note the downward trend in the EC50 - 4PL, which becomes more pronounced over time and the change from Lot 437 and Lot 815.
The x-axis is labeled with the notebook numbers you entered for each run. The data points are ordered according to the acquisition date for each run, which came from the Excel file you imported for each run. Data points are spaced along the x-axis in a fixed increment, so the spacing does not reflect the actual time between runs. The data points are colored according to the analyte Lot Number.Options above the graph allow you to change the scale of the y-axis from linear to logarithmic.
Display Levey-Jennings Plots for Other Performance Metrics
Use the tabs above the Levey-Jennings plot to see charts for:
EC50 - 4PL - the EC50 calculated using a 4-parameter logistic curve and the R transform script
AUC - the area under the fluorescence intensity curve
HighMFI - the highest recorded fluorescence intensity
Click on the AUC and HighMFI tabs to see the trends in those curves as well.
Generate PDFs for Levey-Jennings Plots
If you wish, you can generate a PDF of the plot visible:
Click on the PDF icon in the upper right.
Depending on your browser settings, you may need to allow popups to run.
Explore the Tracking Data Table
Below the graph area, you'll find a table that lists the values of all of the data points used in the Levey-Jennings plots above.
For quicker review of relevant Levey-Jennings plots without generating the full report, you can access them directly from the QC report for your titration or single point control.
Select > Manage Assays, then click Luminex Assay 200.
Select View QC Report > View Titration QC Report.
Click the graph icon in the L-J Plots column of the second row (where the analyte is ENV2 we viewed earlier).
You can select any of the performance metrics from the dropdown. Click EC50 4PL and you can quickly review that Levey-Jennings plot.
Notice that the notebook number for the run whose row we selected (01 in this screencap) is shown in red along the x-axis.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
One component of validating Luminex data is to define a guide set which defines an expected range for the standard for a particular combination of analyte, isotype and conjugate. Each combination may have a different guide set. Once you apply a guide set to a run, the expected ranges are displayed in the Levey-Jennings plots. QC flags will be raised for values outside the given range. Guide sets consist of means and standard deviations for the performance metrics and may be either:
Run-based: calculated from an uploaded set of runs
Value-based: defined directly using known values, such as from historical lab data
You can define multiple guide sets of different types and choose which guide set is applied to any given run. For example, you might define a guide set based on a particular lot of analyte, and use it to check performance of that lot over time, then validate a new lot when analyte preparation or supplier has changed.
Earlier in the Luminex Level II tutorial, we assigned five runs to each lot of analytes, so we can now create different guide sets on this data for each lot of the analyte, one run-based and one value-based. When you later select which guide set to apply, you will be able to see the comment field, so it is good practice to use that comment to provide selection guidance.
Create a Run-based Guide Set
In this tutorial example consisting of just 5 runs per lot, we use the first three runs as a guide set for the first lot. Ordinarily you would use a much larger group of runs (20-30) to establish statistically valid expected ranges for a much larger pool of data.
In the Luminex folder, select > Manage Assays, then click Luminex Assay 200.
Open the Levey-Jennings report for the standard by selecting View QC Report > view levey-jennings reports and clicking Standard1.
Under Choose Graph Parameters, select the following:
Antigens: ENV2
Isotype: IgA
Conjugate: Biotin
Click Apply.
Above the graph, notice that the Current Guide Set box reads: "No current guide set for the selected graph parameters."
Click New next to this message to Create Guide Set.
Notice in the upper right corner of the popup, Run-based is selected by default.
In the All Runs panel, scroll down and click the button next to each of the Assay IDs that begin with 01, 02, and 03 to add them to the guide set.
Enter the Comment: "Guide set for Lot 437"
Click Create.
Notice that the calculated expected ranges are shown applied to the runs you selected as part of the Guide Set. The mean is the average value, the colored bars show the calculated standard deviation. The expected range is three times the standard deviation over or under the mean.Once you define run-based guide sets for a standard, expected ranges are calculated for all performance metrics (AUC and HighMFI), not just EC50 4PL. Switch tabs to see graphed ranges for other metrics.
Create a Value-based Guide Set
If you already have data about expected ranges and want to use these historic standard deviations and means to define your ranges, you create a value-based guide set. Here we supply some known reasonable ranges from our sample data.
Above the graph, click New to Create Guide Set.
You can only edit the most recent guide set, so you will be warned that creating this new set means you can no longer edit the guide set for Lot 437.
Click Yes.
Click Value-based under Guide Set Type.
Enter values as shown:
Metric
Mean
Std.Dev.
EC50 4PL
3.62
0.2
AUC
70000
1000
High MFI
32300
200
Enter the Comment: "Guide set for Lot 815"
Click Create.
Since this guide set is not based on any runs, you will not see expected ranges displayed in the report until it is applied.
Apply Guide Sets
For each run, you can select which guide set to apply. Once a run has a guide set applied, you may switch the association to another guide set, but may not later entirely dissociate the run from all guide sets through the user interface.
Apply Run-based Guide Set
Since we used three of our runs for the first analyte lot, we are only able to apply that guide set to the other two runs from the same Lot 437.
At the bottom of the Levey-Jennings plot, click the checkboxes next to the the runs that begin with the digits 04 and 05.
Click Apply Guide Set.
In the popup, notice that you can see the calculated run-based thresholds listed alongside those you entered for the value-based set.
Select the run-based "Guide set for Lot 437", then click Apply Thresholds.
In the Levey-Jennings plot, observe the range bars applied to run 04 and run 05.
Notice that results for both run 04 and run 05 fall outside of the expected range. We will discuss the QC flags raised by this occurrence in a future step.
Apply Value-based Guide Set
No runs were used to create this set, so we can apply it to all 5 runs that used the second analyte lot.
At the bottom of the Levey-Jennings plot, select the checkboxes next to runs 06-10.
Click Apply Guide Set.
In the popup, make sure that the guide set with comment Guide set for Lot 815 is checked.
Click Apply Thresholds.
Notice that three of the runs from the second lot including values within our ranges, but two fall outside them.Explore the guide sets as displayed in the graphs on the other performance metric tabs for AUC and High MFI.
Manage Guide Sets
You can view the ranges defined by any guide set by selecting View QC Report > view guide sets and clicking Details. Note that for run-based guide sets, only the calculated range values are shown in the popup. Clicking a Graph link will navigate you to the Levey-Jennings plot where the set is defined.
Change Guide Set Associations
Select checkboxes for runs below the Levey-Jennings plot and click Apply Guide Set. You may choose from available guide sets listed. If any selected runs are used to define a run-based guide set, they may not have any guide set applied to them, and requests to do so will be ignored.
Edit Guide Sets
Only the most recently defined guide set is editable. From the Levey-Jennings plot, click Edit next to the guide set to change the values or runs which comprise it. For run based guide sets, use the plus and minus buttons for runs; for value-based guide sets, simply enter new values. Click Save when finished.
Delete Guide Sets
Over time, when new guide sets are created, you may wish to delete obsolete ones. In the case of run-based guide sets, the runs used to define them are not eligible to have other guide set ranges applied to them unless you first delete the guide set they helped define.
Select View QC Report > view guide sets.
Check the box for the obsolete guide set. To continue with the tutorial, do not delete the guide sets we just created.
Click (Delete).
For each guide set selected, you will be shown some information about it before confirming the deletion, including the set of runs which may still be using the given guide set. In this screencap, you see what the confirmation screen would look like if you attempted to delete an old run-based guide set. Value-based sets will only have user runs.
Click Cancel and then either Graph link to return to the Levey-Jennings plot associated with that guide set.
When a guide set is deleted, any QC flags raised by the expected range it defined will be deleted as well.
View QC Flags
When guide sets are applied, runs whose values which fall outside expected ranges are automatically flagged for quality control. You can see these tags in the grid at the bottom of the Levey-Jennings page.
Look at the Standard1 Tracking Data grid below the plots. You can use the grid customizer to control which columns are displayed.
Notice red highlighting is applied to any values that fall out of the applied guide set ranges.
Observe that QC flags have been added for each metric where there are out of range values on each run.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Plotting standard curves for several Luminex® runs together helps visualize any inconsistencies in data and curve fits between runs. The resulting overlay plot is sometimes called a curve "graveyard."Here, we generate an overlay plot for the 4pl standard titration curves for the same data used in the previous tutorial steps.
Steps
If you navigated away after the last tutorial step, return to the Levey-Jennings plot for the ENV2 standard
Track Single-Point Controls in Levey-Jennings Plots
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
LabKey Server can generate a Levey-Jennings plot for a single point control, useful in antigen panels when the control is not titrated. These plots can be used to track quality and performance over time, just as in the case of Step 5: Track Analyte Quality Over Time.The single point plots use the FI-Bkgd data that is uploaded with the run. When the control has two records (i.e. two wells, run in duplicate), the average value is calculated and reported as MFI (Mean Fluorescence Intensity).
Upload Run with Single Point Control Tracking
When you upload an assay run, the Define Well Roles panel lists the controls available for single point tracking. To briefly explore this feature, you can reimport three of the runs we used for the standard Levey-Jennings plots. Repeat these steps for any three runs:
In the Luminex folder, confirm you are logged in.
In the Assay List, click Luminex Assay 200.
Select a row for a run you imported, and click Reimport Run.
As you reimport the run, the values you entered originally are provided as defaults, making it easier to change only what you intend to change.
Leave Batch Properties unchanged and click Next.
Leave Run Properties unchanged and click Next.
In Define Well Roles check the box for IH5672, the only single point control available in this sample data.
Leave Analyte Properties unchanged and click Save & Finish.
Repeat for at least two more runs.
View Levey-Jennings Plots for Single Point Controls
Select View QC Report > view single point control qc report.
Click the Graph link next to any row.
In this example, there are two tracked controls from Lot 815 and 5 from Lot 437. Your graph may vary depending on which runs you re-imported and which report row you selected.
Notice the Average Fi Bkgd column in the lower right. This value is the computed average of the FI-Bkgd value for the two wells, also known as MFI in how the graph is labelled.
As with titrated standards, you can select the Graph Parameters, antigen, isotype, and conjugate. You can also define guide sets and raise qc flags for values of the single point controls which fall outside the expected range. Find more information in Step 6: Use Guide Sets for QC.
View Levey-Jennings Plots from QC Reports
For quicker review of the single point control Levey-Jennings plots, you can access them directly from the QC report.
Select View QC Report > View Single Point Control QC Report.
Click the graph icon in the L-J Plots column of any row.
You can quickly review the plot without visiting the full page.
The notebook number for the row you selected is shown in red along the x-axis. In this image, notebook 01.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
A key component of using Luminex® instrument data is calculation of logistic curve fits as well as other values for standards, unknowns, and quality controls. During the second Luminex tutorial, in Step 4: View 4pl Curve Fits, you saw the 4pl logistic curve fits, and then used some of the calculated values in Step 5: Track Analyte Quality Over Time.Some calculations are done by the BioPlex software itself, LabKey Server performs others, and still more are made using R and various packages by way of the LabKey Luminex transform script. By further customizing the assay design and adding additional operations to the transform script, additional calculations may be added enabling you to tailor the instrument data framework to suit your specific research needs.
The LabKey Luminex transform script uses R packages to calculate logistic curve fits for each titration for each analyte. Titrations may be used either as standards or as quality controls. In this tutorial, all titrations are used as standards.Curve fits are calculated using a 4 parameter logistic (4pl). For each run, the script outputs a PDF that includes plots for curve fits for each analyte. Each plot shows the dose response curve for fluorescence intensity with increasing concentration or reduced dilution. These plots can be useful for examining how well the curves fit the data.
LabKey Server Calculations
LabKey Server itself calculates the AUC ("Area Under the Curve") for each standard titration using a trapezoidal calculation based on observed values. LabKey Server also identifies the HighMFI ("Highest Mean Fluorescence Intensity") for each titration.
BioPlex Calculations vs. LabKey Luminex Transform Script Calculations
The Excel run files produced by the BioPlex instrument also include 5-parameter logistic curve fits for each titrated standard and each titrated quality control. These 5pl regressions were calculated by the BioPlex software. The Excel run files also include estimates for sample concentrations generated using each sample's measured FI-Bkgd (fluorescence intensity minus background well fluorescence intensity) and the 5pl regression equations derived for the standards by the instrument software.
Review Calculated Values
In Step 4: View 4pl Curve Fits you reviewed calculated values as they are available in both titration qc reports or in the results grid for the run data. The titration qc report includes summary results for both standards and QC controls for a given run or for all runs. The results grid includes regression parameters for all curve fits.
Subtract Negative Control Bead Values
The FI-Bkgd-Neg column shows the fluorescence intensity after both the background well and the negative bead are subtracted. The assay design used in the Luminex Assay Tutorial Level II tells the script (via the StandardCurveFitInput property's default value) to use fluorescence alone (FI), without subtraction, to fit 4pl curves to titrated standards. In contrast, the assay design tells the script (via the UnknownFitInput property's default value) to use FI-Bkgd-Neg to estimate concentrations for unknowns using the 4pl regression equations calculated from the standards.When calculating the value FI-Bkgd-Negative (fluorescence intensity minus background FI minus a negative bead), you may specify on a per analyte basis what to use as the negative bead. Depending on the study, the negative bead might be blank, or might be a more suitable negative control antigen. For example, in a study of certain HIV antigens, you might subtract MulV gp70 from gp70V1V2 proteins, blank from other antigens, etc. Note that the blank bead is not subtracted by default - it must be explicitly selected like any other negative bead.To enable subtraction of negative control bead values, the assay design must be modified to include a run field of type Boolean named NegativeControl. Then during the assay run import, to select negative beads per analyte, you'll set Analyte Properties when importing data files. First identify specific analytes using checkboxes in the Negative Control column, then select one of these negative beads for each analyte in the Subtract Negative Bead column.
Use Uploaded Positivity Threshold Defaults
If your lab uses specific positivity cutoff values, you can manually enter them on an antigen-by-antigen basis during upload on the Analyte Properties panel. To simplify user entry and reduce the possibilities of errors during this process, you may specify analyte-specific default values for the PositivityThreshold property on a per folder and assay design combination. The default default value is 100. To specify analyte-specific defaults, add them to the assay design for a given folder as described here using the Luminex Assay Tutorial Level II example:
Select > Manage Assays, then click Luminex Assay 200.
Enter the Analyte and desired Positivity Threshold.
Click Add Row to add another.
You may instead click Import Data to upload or paste TSV file data to simplify data entry.
Click Save Defaults when finished.
When you import a set of positivity threshold data, it overwrites the prior set, meaning that any defaults previously defined but missing from the imported TSV will be dropped.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
When guide sets are applied, runs whose values fall outside expected ranges are automatically QC flagged. You can see these flags in the grid at the bottom of the Levey-Jennings page. This walkthrough uses the Luminex Tutorial Level II results. You can also see flags in the interactive example
From the Runs grid, select View QC Report > View Levey-Jennings Reports and click Standard1.
Select ENV2, IgA, and Biotin as the Antigen, Isotype, and Conjugate respectively. Click Apply.
Look at the Standard1 Tracking Data grid below the plots.
Observe red highlighting applied to values in the Four Parameter Curve Fit EC50, HighMFI, and Trapezoidal Curve Fit AUC columns. View the plot for each metric to see the points which lie outside the guide set ranges.
Observe how QC flags have been added for each metric showing out-of-range values for each run.
Note that the QC flags are specific to the combination of antigen, isotype and conjugate parameters you selected.
Deactivate QC Flags
It is possible to deactivate a single QC flag, in this case we will disable the flag for EC50 - 4PL for the tenth run. When you deactivate a flag manually in this manner, it is good practice to add an explanatory comment.
Click on the QC flags (AUC, EC50-4) for the run that begins with "10".
You now see the Run QC Flags popup.
In the Enabled column, uncheck the box for the EC50-4 flag. This deactivates, but does not delete, the associated QC flag for this data. Notice the red triangle in the corner indicating a change has been made.
Click the Comment region for that row and type "Manually disabled."
Click Save in the popup.
In the Standard1 Tracking Data grid, notice two changes indicating the deactivated flag:
In the QC Flag column for run 10, the EC50-4 QC flag now has a line through it.
In the EC50 4pl column for run 10, the value is no longer highlighted in red.
If your run-based guide set does not have enough valid data for a given metric to usefully flag other runs, you might choose to disable all QC flagging based on that metric. In an extreme example, if only one run in the guide set had a valid EC50-4PL value, then the standard deviation would be zero and all other runs would be flagged, which isn't helpful in assessing trends.
Select View QC Report > View guide sets, then click the Details link for the guide set you want to edit. In this tutorial, the only run-based guide set is the one we created for Lot 437.
The Num Runs column lists the number of runs in the guide set which have valid data for the given metric. In this example all three runs contain valid data for all metrics.
To disable QC flagging for one or more metrics, uncheck Use for QC boxes.
Click Save.
There is also a details button on the active guide set in the Levey-Jennings plot UI giving you quick access to its parameters. Only the most recently defined guide set is editable.Disabling QC flagging is an exception, as long as the guide set has been applied to other runs before defining a new guide set, you can return later to enable or disable flagging as needed.
View QC Flags in the QC Report
The same red highlighting appears in the QC Reports available for the entire assay and for each individual run. When you view the QC report for the entire assay, you see red highlighting for all out-of-range values detected using guide sets, not just the ones for a particular analyte (as in the steps above). Red highlighting does not appear for any value whose QC flag has been manually deactivated.Here, we view the QC report for the entire assay, then filter down to see the results matching the selections we made above.
Click View QC Report > View titration qc report.
Observe a grid that looks like this one in the interactive example.
Filter the grid so you see only the rows associated with the same analyte/isotype/conjugate combination that was associated with guide sets in earlier steps. Our example data only has one isotype and conjugate, so further filtering to those selections is unnecessary here:
Click the Analyte column header.
Choose Filter.
Select ENV2.
Click OK.
You can now see the same subset of data with the same red highlighting you saw on the Levey-Jennings page.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Working through the two Luminex Tutorials gives you a broad overview of many of the features available in the LabKey Server Assay Tools for working with this instrument data. This topic consolidates additional detail and information about features and properties used with this technology.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
This topic reviews some details and built in features of the Luminex Assay 200 design you uploaded in archive form during the Luminex Level II tutorial.
Explore Lookups and Defaults in the Assay Design
Open the assay design and examine how lookups and default values are included as part of this assay design to simplify and standardize data entry:
Select > Manage Assays, then click Luminex Assay 200
Click Manage Assay Design > Edit assay design.
Click the heading to open the Run Fields section.
Expand the Isotype field.
Click Advanced Settings for the Isotype field.
Note the Default Value of "IgA" has been set for this lookup property.
When finished reviewing, be sure to exit with Cancel to discard any changes you may have made.
Note the following in the above screenshot:
Lookup Definition Options:
The Data Type for Isotype is a lookup to the Isotype list.
User-facing result: When importing runs, users will be shown a dropdown list of options for this field, not a free-form data entry box. The options will be the values on the Isotype list.
Default value
In the Advanced Settings and Properties popup, you can see that the initial Default value for the Isotype field has been set to IgA.
User-facing result: When users first import runs, the list of dropdown options for Isotype will show a default of IgA. Choosing the default appropriately can speed data entry for the common case.
Default Type
In the Advanced Settings and Properties popup, you can also see that the Default Type for the Isotype field has been pre-set to Last entered
User-facing result: When a user imports a run, the list of dropdown options will default to the user's "last-entered" value. If the user selected "IgB" for their first run, the next will default to "IgB" instead of the original default value of "IgA".
You can use steps similar to the ones above to explore other fields in the assay design (e.g., Conjugate) to see how lookups and/or defaults for these fields are pre-configured.
Review Assay Properties
While you have the assay design open, you may want to review in more detail the properties defined. See further details in Luminex Properties. As with any assay design, an administrator may edit the default design's fields and defaults to suit the specific needs of their data and project. To do so, it is safest to make changes to a copy of the assay design if you want to still be able to use the tutorial. To copy a design:
Choose the assay you want to copy from the Assay List.
Select Manage Assay Design > Copy assay design.
Choose a destination folder or click Copy to Current Folder.
Give the new copy a new name, change fields and properties as required.
Click Save when finished.
The LabKey Luminex transform script requires certain fields to exist in the assay design in order for it to have locations to place its results. In the tutorial we make a small edit to the base design to configure the Transform Script, but avoid making other changes or the tutorial may not work.For further information on setting up a custom assay that includes the fields used by the transform script, see:
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Default Luminex® assay designs include properties specific to the technology, and beyond the default properties included in General assay designs. The general process for designing an assay is described in Design a New Assay. A design based on the built-in defaults for Luminex is defined in the Luminex assay tutorials (Step 1: Create a New Luminex Assay Design). This page offers additional details on the default properties defined for this type of assay.
For optional, additional properties that are used by the Luminex QC transform script, see below, or review Review Fields for Script.
Assay Properties
Import in Background
Using this option is particularly helpful when you have large runs that are slow to upload. If this setting is enabled, assay uploads are processed as jobs in the data pipeline.You will see the Upload Jobs page while these runs are being processed. Your current jobs are marked "Running." When the jobs have completed, you will see "Completed" instead of "Running" for the job status for each. If you see "Error" instead of completed, you can see the log files reporting the problem by clicking on "Error." Luminex assay properties (batch, run, well role/titration settings, and analyte properties) are also written to the log file to assist in diagnosing upload problems.When the Status of all of your jobs is "Completed", click the Description link for one of the runs to see all of the data for this run.
Transform Scripts
Add any transform script here by clicking Add Script, then either uploading the script or specifying the path. During script development, you can also check the box to Save Script Data for Debugging.Learn more in this topic:
If you want the assay to Auto-Link Data to Study, select the target study here. You can also choose the Linked Dataset Category if desired.Learn more in this topic:
The user is prompted for batch properties once for each set of runs during import. The batch is a convenience to let users set properties once and import many runs using the same suite of properties.Included by default:
Participant Visit Resolver: Required. This field records the method used to associate the assay with participant/visit pairs. The user chooses a method of association during the assay import process.
TargetStudy: Including this field simplifies linking assay data to a study, but it is not required. Alternatively, you can create a property with the same name and type at the run level so that you can then link each run to a different study.
Network: Enter the network.
Species: Enter the species under study.
LabID: The lab where this experiment was performed.
Analysis Software: The software tool used to analyze results. For LabKey's Luminex tool, this is typically "Bioplex."
Run Properties
The user is prompted to enter run level properties for each imported file. These properties are used for all data records imported as part of a Run.Included by default:
Assay Id: If not provided, the file name is used.
Comments
Isotype
Conjugate
Test Date
Replaces Previous File: Boolean
Date file was modified
Specimen Type
Additive
Derivative
Run Data (Required): Data files must be in the multi-sheet BioPlex Excel file format.
Well Roles
The user is prompted to select well roles next. If a sample appears at several different dilutions, we infer you have titrated it. During the import process for the run, you can indicate whether you are using the titrated sample as a standard, quality control, or unknown.Included by default:
Well Roles for Standard, QC Control, and Other Control
Single Point Controls
A standard is typically the titration used in calculating estimated concentrations for unknowns based on the standard curve. A quality control is typically a titration used to track values like AUC and EC50 over time for quality purposes. Learn more in Luminex Calculations.In this panel you will also see checkboxes for Tracked Single Point Controls. Check a box if you would like to generate a Levey-Jennings report to track the performance of a single-point control. Learn more in the Level II tutorial which includes: Step 5: Track Analyte Quality Over Time and Track Single-Point Controls in Levey-Jennings Plots.
Analyte Properties
On the same page as well roles, the Analyte Properties section is used to supply properties that may be specific to each analyte in the run, or shared by all analytes in the run. They are not included in the data file, so need to be entered separately. For example, these properties may help you track the source of the analytes or beads used in the experiments, for the purpose of quality control. In the second Luminex tutorial we track two lots of analytes using these properties.For each analyte the user enters (a checkbox allows entering the same value for all analytes for any property):
Standard Name
Analyte Type
Weighting Method
Bead Manufacturer
Bead Dist
Bead Catalog Number
Use Standard: Elect whether to use the standard specified under well roles as the standard for a given analyte.
Additional analyte properties present in the data:
PositivityThreshold: The positivity threshold.
AnalyteWithBead: The name of the analyte including the bead number.
BeadNumber: The bead number.
Excel File Run Properties
When the user imports a Luminex data file, the server will try to find these properties in the header and footer of the spreadsheet, and does not prompt the user to enter them.Included by default:
The Excel file format includes a single field for sample information, the Description field. LabKey Server will automatically attempt to parse the field to split it into separate fields if it contains information like Participant and Date. Formats supported include:
The "Visit" before the visit number itself is optional. <Date> and <ExtraInfo> are optional as well. LabKey Server will use the value of the "TargetStudy" field (specified as a batch or run field), if present, to resolve the SpecimenId.
Results/Data Properties
The user is prompted to enter data values for each row of data associated with a run.Not included by default in the design, but should be considered:
SpecimenID: For Luminex files, data sources are uniquely identified using SpecimenIDs, which in turn point to ParticipantID/VisitID pairs. For Luminex Assays, we automatically extract ParticipantID/VisitID pairs from the SpecimenID. If you exclude the SpecimenID field, you will have to enter SpecimenIDs manually when you copy the data to a study.
Additional Properties for the Transform Script
The LabKey Luminex transform script calculates additional values (e.g., curve fits and negative bead subtraction) that are used by the LabKey Luminex tool. Custom batch, run, analyte, and data properties used by this script are covered in these pages: Customize Luminex Assay for Script and Review Fields for Script. Some useful assay properties are listed here:Assay Properties
Field Label
Value
Description
Editable Runs
Unchecked
When selected, allows run data to be edited after import by default. If you allow editing of run data, you may wish to uncheck Advanced Settings > Display Options > Show on update form... in the domain editor for each field used or calculated by the script. The script runs only on data import, so preventing later editing of such fields is necessary for calculated data to continue matching the values displayed for the fields in the assay.
Import in Background
Unchecked
When selected, runs are imported in the background, allowing you to continue work on the server during import. This can be helpful for importing large amounts of data. This tutorial leaves this value unchecked merely for simplicity of workflow. For further information on what happens when you check this property, see Luminex Properties.
Transform Script
--
Path to the LabKey Luminex transform script. The path provided must be local to your server. The default path provided in a XAR will be usable only on the server where the XAR was created.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
LabKey Server understands and processes Excel files of Luminex® results that have been output by a Bio-Plex instrument's software for Luminex experiments. You can see an example Excel file here. This page reviews the Excel file format.
Microsoft Excel files in *.xls (1997-2003) and *.xlsx (2007-2013) formats are supported; Microsoft Excel *.xls 95/5.0 format is not supported.
LabKey Server's Luminex features have been tested using data for 96-well plates, but are expected to work for 384-well plates as well. The larger plates are used by newer instruments that allow multiplexing of up to 500 analytes in each well. Devices that only support 96-well plates are usually limited to a maximum of 100 analytes in each well.
General Characteristics
The file is typically a multi-sheet workbook
Each spreadsheet (tab) in the workbook reports data for a single analyte. One sheet may report measurements for the blank bead.
Each sheet contains:
A header section that reports run metadata (see below for details)
A data table that reports values for wells
A footer section that describes data flags and shows curve fits equations and parameters
Run Metadata in the Header
Most of the metadata fields reported in the file header are imported as "Excel File Run Properties." With the exception of Analyte, all of these metadata properties are the same for each sheet in the workbook. Analyte is different for each sheet.
File Name - Imported
Analyte - The analyte name is also the name of the worksheet (the tab label).
Acquisition Date - Imported
Reader Serial Number - Imported
Standard Lot - Ignored in the header. The standard name also appears in the description for wells that contain the standard. The name of the standard for each analyte can be entered during the import process for each run.
Expiration Date - Ignored
Plate ID - Imported
Signed By - Ignored
Document ID - Ignored
RP1 PMT (Volts) - Imported
RP1 Target - Imported
Well data table
The data table in the middle of each sheet shows how each sample in each well reacted with the single analyte reported on the sheet. In other words, every well (and every sample) appears on every sheet, but each sheet reports on the behavior of a different analyte within that well.
File format variants
Samples (particularly unknowns) are typically replicated on a plate, in which case they will appear in multiple wells. How these replicates appear in the Excel file depends on the file format. LabKey Server understands three types of file formats.
Variant A - Summary Data:
The data table contains one row per sample. This row provides a calculated average of observed values for all sample replicates (all wells where the sample was run).
The Wells column lists all the wells used to calculate the average.
Variant B - Raw Data:
The data table contains one row per well. That means that each sample replicate appears on its own line and the data reported for it is not averaged across sample replicates. Consequently, there are multiple rows for each experimental sample.
Here, the Wells column shows only a single well.
LabKey Server infers the presence of Variant B format if a files shows multiple rows for the same sample (so replicates are on different lines). Usually, all samples have the same number of replicates.
Variant C - Summary and Raw Data:
Two data tables appear, one following Variant A formatting and the second following Variant B. In other words, tables for both summary sample data and for individual well data both appear.
The data used in the tutorial follows the Variant B format. Two replicates were run for each unknown sample, so each unknown sample appears on two lines, one for each well where it was run.
Data columns
The columns in the well data table:
Analyte
The analyte name is also the name of the worksheet (the tab label). In the data used in this tutorial, the analyte name is a combination of the analyte name and the number of the bead bound to the analyte.
Type
The letter portion of the "Type" indicates the kind of sample; the number portion (if there is one) provides a sample identifier
B - Background. The average background fluorescence observed in a run's background wells is subtracted from each of the other wells to give the “FI - Bkgd” column
S - Standard. The titration curve for the standard is used to calculate the concentration of unknowns
C - Quality control. Used to compare with other runs and track performance
X - Unknown. These are the experimental samples being studied.
Well
The plate location of the well where the sample was run. For file format Variant A, multiple wells may be listed here - these are all the wells for the sample replicate that have been averaged together to produce the average fluorescence intensity for the sample, as reported in FI
Description
For an unknown, identifies the source of the unknown sample. This may be a sample identifier or a combination of participant ID and visit number. The following formats are supported, and checked for in this sequence:
<SpecimenID> - The value is checked for a matching specimen by its unique ID in the target study. If found, the specimen's participant ID and visit/date information is stored in the relevant fields in the Luminex assay data.
<SpecimenID>: <Any other text> - Any text before a colon is checked for a matching specimen in the target study. If found, the specimen's participant ID and visit/date information is stored in the relevant fields in the Luminex assay data.
<SpecimenID>; <Any other text> - Any text before a semi-colon is checked for a matching specimen in the target study. If found, the specimen's participant ID and visit/date information is stored in the relevant fields in the Luminex assay data.
<ParticipantId>, Visit <VisitNumber>, <Date>, <ExtraInfo> - The value is split into separate fields. The "Visit" prefix is optional, as is the ExtraInfo value. The VisitNumber and Date values will be ignored if they cannot be parsed as a number and date, respectively.
FI
For file format Variant B, this is the median fluorescence intensity observed for all beads associated with this analyte type in this well.
For file format Variant A, this value is the mean median fluorescence intensity for all sample replicate wells; in other words, it is a mean of all median FI values for the wells listed in the Wells column.
FI-Bkgd
Fluorescence intensity of well minus fluorescence intensity of a "background" (B) well
Std Dev
Standard of deviation calculated for replicate wells for file format Variant A
%CV
Coefficient of variation calculated for replicate wells for file format Variant A
Obs Conc
Observed concentration of the titrated standard, quality control or unknown calculated by the instrument software. Calculated from the observed FI-Bkgd using the 5-parameter logistic regression that is reported at the bottom of the page as Std. Curve.
Indicators: *Value = Value extrapolated beyond standard range; OOR = Out of Range; OOR> = Out of Range Above; OOR< = Out of Range Below
Values of Obs Conc flagged as *Value and OOR receive substitutions (to more clearly show ranges) during import to LabKey Server. See: Luminex Conversions
Exp Conc
Expected concentration of the titrated standard or quality control. Known from the dilutions performed by the experiment performer. The Std. Curve equation listed at the bottom of the page reports the 5 parameter logistic regression equation that the instrument software fit to the distribution of the titration's FI-Bkgd and Exp Conc. The FitProb and ResVar for this regression are also listed.
(Obs/Exp)*100
100 times the ratio of observed and expected concentrations for titrated standard or quality control
Indicators: *** = Value not available; --- = Designated as an outlier
Group
Not used in this tutorial
Ratio
Not used in this tutorial
Dilution
Dilution of the sample written as an integer. The actual dilution is a ratio, so a dilution of 1:100 is noted as 100
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
In the Luminex Assay Tutorial Level II, we import a batch of runs but gloss over the Define Well Roles section of import. The checkboxes there are used to identify titrations and single point controls you want to be able to use later. Return to this view by beginning to import (or reimport) any run. After entering run properties, you will see:
Standard vs. Quality Control
In the Define Well Roles section, you can mark a titration as providing both a standard and a quality control. If you check both, you will see twice as many curves (see Step 4: View 4pl Curve Fits). Standard titrations are used to calculate the concentrations of samples and the values displayed in Levey-Jennings plots (see Step 5: Track Analyte Quality Over Time). In contrast, quality controls are used in a lab-specific manner to track the performance of the assay over time. For this tutorial, we designate our titrated standards as standards, but not as quality controls.
Other Control
In order to have values for EC50, AUC, and other calculations done without adding the selected titration to the Levey-Jennings plot and QC Report, select the Other Control checkbox. This is useful for titrations that will not be tracked using these reporting tools, but still need the transform script to calculate values.
Multiple Standards per Analyte
If the data for your run included multiple standard titrations, you would be able to choose which analytes to associate with which standards. Note that you must have marked each standard as a standard in the Define Well Roles section before it will appear as an option. Each standard will appear over a column of checkboxes in the Analyte Properties section. You can select more than one standard for each analyte by selecting the checkboxes in more than one standard’s column. The data for this tutorial includes only one standard, so this option is not explored here.
Single-Point Controls
To track performance of a single-point control over time, such as in the case of an antigen panel where the control is not titrated, you can select Tracked Single Point Controls. Check the appropriate boxes in the Define Well Roles panel during run upload. This feature is explored in Track Single-Point Controls in Levey-Jennings Plots.
Cross-Plate Titrations
During import of Luminex data, if a sample appears at different dilutions, we infer that it has been titrated and the user can indicate whether they are using the titrated sample as a standard, quality control, or unknown.When multiple plates of data are uploaded together in a single run, LabKey will infer cross-plate titrations when either of the following conditions are met:
If the value in the Description field is repeated 5 or more times in the "Summary" section, or
If the Description is repeated 10 or more times in the "Raw" section.
These identified potential cross-plate titrations are assigned a name matching the description field that identified them, and the user confirms during upload if they are indeed titrations using a checkbox. Uncheck any incorrect inferrals.The titration data is then used for result calculations and further analysis and plots.
Premium Feature - Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
During upload of Luminex files, we perform substitutions for certain flagged values. Other types of flagged values are imported without alteration.
Substitutions During Import for *[number] and OOR
We perform substitutions when Obs. Conc. is reported as OOR or *[number], where [number] is a numeric value. *[number] indicates that the measurement was barely out of range. OOR< and OOR> indicate that measurements were far out of range.
To determine the appropriate substitution, we first determine the lowest and highest "valid standards" for this analyte using the following steps:
Look at all potentially valid standards for this run. These are the initial data lines in the data table on the Excel page for this Analyte. These lines have either “S” or “ES” listings as their types instead of “X”. These are standards (Ss) instead of experimental results (Xs). Experimental results (Xs) are called Wells in the following table.
Determine validity guidelines. Valid standards have values in the (Obs/Exp) * 100 column that fall “within range.” The typical valid range is 70-130%, but can vary. The definition of “within range” is usually included at the end of each Excel page on a line that looks like: “Conc in Range = Unknown sample concentrations within range where standards recovery is 70-130%.” If this line does not appear, we use 70-130% as the range.
Now identify the lowest and highest valid standards by checking the (Obs/Exp) * 100 column for each standard against the "within range" guideline.
N.B. The Conc in Range field will be *** for values flagged with * or OOR.
In the following table, the Well Dilution Factor and the Well FI refer to the Analyte Well (the particular experiment) where the Obs. Conc. was reported as OOR or as *[number].
When Excel Obs. Conc. is...
We report Obs. Conc. as...
Where [value] is...
OOR <
<< [value]
the Well Dilution Factor X the Obs. Conc. of the lowest valid standard
OOR >
>> [value]
the Well Dilution Factor X the Obs. Conc of the highest valid standard.
*[number] and Well FI is less than the lowest valid standard FI
< [value]
the Well Dilution Factor X the Obs. Conc. of the lowest valid standard.
*[number] and Well FI is greater than the highest valid standard FI
> [value]
the Well Dilution Factor X the Obs. Conc of the highest valid standard.
If a valid standard is not available (i.e., standard values are out of range), [value] is left blank because we do not have a reasonable guess as to the min or max value.
Flagged Values Imported Without Change
Flag
Meaning
Column
---
Indicates that the investigator marked the well(s) as outliers
Appears in FI, FI Bkgd and/or Obs. Conc.
***
Indicates a Machine malfunction
Appears in FI, FI Bkgd, Std. Dev, %CV, Obs. Conc., and/or Conc.in Range
[blank]
No data
Appears in any column except Analyte, Type and Well, Outlier and Dilution.
Customize Luminex Assay for Script
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
This topic details the minimal number of setup steps necessary to create a custom Luminex assay that works with the LabKey Luminex transform script. You can start from the default Luminex assay type that is built into the server, or as an alternative, it may be easier to start with the XAR-generated assay design used by the Luminex Assay Tutorial Level II and customize the design to your needs.It may be helpful to use the Review Fields for Script page in addition to this topic.
Create a New Assay Design
These steps assume that you are in a "Luminex" folder (of type "Assay"), as described in Set Up Luminex Tutorial Folder.
Select > Manage Assays.
Click New Assay Design.
Click the Specialty Assays tab.
Under Use Instrument Specific Data Format, choose Luminex.
For the Assay Location, choose Current Folder (folder name). This is important to ensure that lookups to lists in the same folder will work.
Click Choose Luminex Assay.
Give the new assay design a Name.
Optional: Check the Import in Background box.
Checking this box means that assay imports will be processed as jobs in the data pipeline, which is helpful because Luminex runs can take a while to load.
Add fields as described in the following sections before clicking Save in the bottom right.
Add Script-specific Fields
Make sure not to put a space in the Name property for any field you add. See Review Fields for Script page for details about each field.Advanced settings and properties let you customize field behavior. For example, you can hide, and thus prevent editing of particular values after a run has been imported - in cases where those values are used for calculations/graphs during import. For example, we don't want the user inadvertently changing the script version each time they run it. It should only change when the version actually changes.You can also use Default Value Options to guide your users with smart defaults on a per-folder basis.
Add Batch Fields
In the Batch Fields section, click the Add Field button for each of the following:
Add a field for TransformVersion, expand it, then click Advanced Settings.
Optional: Uncheck the Shown on insert… and Show on update… options (so that the user is not asked to enter a value).
Optional: Change the Default Type to Fixed Value (so that you can specify a fixed value for the default for this field).
Add Run Fields
In the Run Fields section:
Add a field for NotebookNo of type Text.
Add a field for SubtNegativeFromAll of type Boolean.
Optional: In the Advanced Settings, uncheck the Show on update.. and Show on insert... boxes.
Add a field for StndCurveFitInput:
The type of this field can be either Text or a lookup to a list which has the following three string values: FI, FI-Bkgd, FI-Bkgd-Neg.
Optional: In the Advanced Settings, uncheck the Show on update.. and Show on insert... boxes.
Optional: based on the lab preference, you may want to set the Default Type for this field to either remember the last entered value or to have an editable default value selected (the script defaults to using the “FI” column if no value is specified for the StndCurveFitInput field). When creating this lookup and others, you may find it useful to import the list archive provided in the Set Up Luminex Tutorial Folder. If you import this archive of lists into the same folder as your assay, you can set this field to target the relevant list.
Add a field for UnkCurveFitInput:
The type of this field can be either Text or a lookup to a list which has the following three string values: FI, FI-Bkgd, FI-Bkgd-Neg.
Optional: In the Advanced Settings, uncheck the Show on update.. and Show on insert... boxes.
Optional: Based on the lab preference, you may want to set the Default Type of this field to either remember the last entered value or to have an editable default value selected (the script defaults to using the “FI” column if no value is specified for the UnkCurveFitInput field)
Optional: Add a field for CalculatePositivity of type Boolean.
Optional: Add a field for BaseVisit of type Decimal.
Optional: Add a field for PositivityFoldChange of type Decimal.
Add Analyte Properties
In the Analyte Properties section:
Add a field for LotNumber of type Text.
Optional: Add a field for NegativeControl of type Boolean.
Add Data Fields
In the Assay Data Fields section:
Add a field for FIBackgroundNegative of type Decimal.
Optional: Add a field for Positivity of type Text.
Optional: Add the optional Data Property fields listed in Appendix D of Review Fields for Script. These are filled in by the transform script and may be interesting to statisticians.
Once all of the custom properties have been added to the assay design, click Save in the lower right.
Customize Data Grids
Any properties you add to the assay design can also be added to the various results, run, and batch grid views for the assay using the (Grid Views) > Customize Grid menu option.
Premium Feature - Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Custom Assay Fields for LabKey Luminex Transform Script
To set up a Luminex assay to run the LabKey Luminex transform script used in the Luminex Assay Tutorial Level II, you need to include certain custom fields in the assay design. The script outputs results into these fields. This page provides details on these output fields.
For reference, the fields included by default in a Luminex assay design are listed on the Luminex Properties page.
Appendix A: Custom Batch Fields
Name
Label
Type
Description
TransformVersion
Transform Script Version
Text
Version number of the transform script (to be populated by the transform script)
Appendix B: Custom Run Fields
Name
Label
Type
Description
SubtNegativeFromAll
Subtract Negative Bead from All Wells
Boolean
Controls whether or not the negative bead values should be subtracted from all wells or just the unknowns. Values for the negative bead for each run are reported on the Negative (Bead Number) tab of the run's Excel file.
StndCurveFitInput
Standards/Controls FI Source Column
Text
The source column to be used by the transform script for the analyte/titration curve fit calculations of Standards and QC Controls (if lookup configured, choices include: FI, FI-Bkgd, and FI-Bkgd-Neg).
UnkCurveFitInput
Unknowns FI Source Column
Text
The input source column to be used by the transform script when calculating the estimated concentration values for non-standards (if lookup configured, choices include: FI, FI-Bkgd, and FI-Bkgd-Neg).
NotebookNo
Notebook Number
Text
Notebook number
AssayType
Assay Type
Text lookup
Lookup into lists.AssayType
ExpPerformer
Experiment Performer
Text
Who performed the experiment
CalculatePositivity
Calculate Positivity
Boolean
Whether or not the calculate the positivity for this run
BaseVisit
Baseline Visit
Decimal
The baseline visit for positivity calculations
PositivityFoldChange
Positivity Fold Change
Integer - lookup with 3x and 5x
Fold change used to determine positivity
Appendix C: Custom Excel File Run Properties
Name
Label
Type
Description
FileName
File Name
Text
The file name
AcquisitionDate
Acquisition Date
DateTime
ReaderSerialNumber
Reader Serial Number
Text
PlateID
Plate ID
Text
RP1PMTvolts
RP1 PMT (Volts)
Decimal
RP1Target
RP1 Target
Text
Appendix D: Custom Analyte Properties
Name
Label
Type
Description
LotNumber
Lot Number
Text
The lot number for a given analyte
NegativeControl
Negative Control
Boolean
Indicates which analytes are to be treated as negative controls (i.e. skip curve fit calculations, etc.)
Appendix E: Custom Data Fields
The fields in this section are optional and not required for the script to run. They may be useful to statisticians.
Name
Label
Type
Description
FIBackgroundNegative
FI-Bkgd-Neg
Decimal
The value calculated by the transform script by subtracting the FI-Bkgd of the negative bead from the FI-Bkgd of the given analyte bead
Positivity
Positivity
Text
The transform script calculated positivity value for unknowns
Slope_4pl
Slope Param 4 PL
Decimal
Optional. The transform script calculated slope parameter of the 4PL curve fit for a given analyte/titration
Lower_4pl
Lower Param 4 PL
Decimal
Optional. The transform script calculated lower/min parameter of the 4PL curve fit for a given analyte/titration
Upper_4pl
Upper Param 4 PL
Decimal
Optional. The transform script calculated upper/max parameter of the 4PL curve fit for a given analyte/titration
Inflection_4pl
Inflection Param 4 PL
Decimal
Optional. The transform script calculated inflection parameter of the 4PL curve fit for a given analyte/titration
Troubleshoot Luminex Transform Scripts and Curve Fit Results
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
This page provides tips on interpreting and fixing error messages from Luminex transform scripts. In addition, it includes troubleshooting advice for issues you may encounter when reviewing assay data and calculated values output from such scripts.
Transform Script Upload Errors
"An error occurred when running the script [script-filename.R] (exit code: 1)"
This message indicates that an error has occurred in the R transform script and has halted the script execution. In most case, if you look further down in the upload log file, you will see the details of the actual R error message.
"Error in library(xtable) : there is no package called 'xtable' - Calls: source -> withVisible -> eval -> eval -> library - Execution halted"
The named library cannot be located. You may need to download an additional package or check that your downloaded packages are in the R library directory. If you are using the R graphical user interface on Windows, you may need to hand copy the downloaded packages from a temp directory into the R/library directory. See the R documentation for more information about troubleshooting R in Windows.
"Illegal argument (4) to SQL Statement: NaN is not a valid parameter" "Zero values not allowed in dose (i.e. ExpConc/Dilution) for Trapezoidal AUC calculation"
When the server attempts to calculate the area under the curve value for each for the selected titrations using the trapezoidal method, it uses the log of the ExpConc or Diliution values. For this reason, zero values are not allowed in either of these columns for the titrations that will have an AUC calculated.
"ERROR: For input string: 1.#INFE+000"
There is at least one bad value in the uploaded Excel file. That value cannot be properly parsed for the expected field type (i.e. number).
"NAs are not allowed in subscripted assignments"
This error has already been fixed to give a better error message. This error is an indication that values in the ExpConc column for some of the wells do not match between Analyte tabs of the Excel file. Verify that the ExpConc and Dilution values are the same across analytes for each of your well groups. Missing descriptions in control wells can also cause this error.
"Error in Ops.factor(analyte.data$Name, analytePtids$name[index])"
This error message indicates that there is a mismatch between the analyte name on the Excel file worksheet tab and the analyte name in the worksheet content (i.e. header and analyte column). Check that the bead number is the same for both.
Issues with uploaded results, curve fit calculations, plots, etc.
Missing values for AUC or EC50
When a curve fit QC metric (such as AUC or EC50) is blank for a given analyte, there are a few reasons that could be the cause (most of which are expected):
Check the curve fit's failure flag to make sure the parameters weren't out of range (e.g. 'AnalyteTitration/FiveParameterCurveFit/FailureFlag')
Check to see if Max(FI) of the curve fit points are less than 1000 - in which case the curve fit won't be run
Check to make sure that the AUC column being displayed is from the 'Trapezoidal Curve Fit' method and EC50 column is from the 'Five Parameter' or 'Four Parameter' fit method
Was the titration selected as QC Control or Standard on upload? (Check 'Well Role' column)
Levey-Jennings report showing too many or too few points on the default graph
The default Levey-Jennings report will show the last 30 uploaded results for the selected graph parameters. You can set your desired run date range using the controls above the graph to view more/less records.
Levey-Jennings Comparison plots and/or Curve Fit PDF plots showing curves sloping in different directions
QC Control titrations are plotted with dilution as the x-axis whereas Standard titrations are plotted with expected concentration on the x-axis. Make sure that your titrations were correctly set as either a QC Control or Standard on the well-role definition section of the Luminex upload wizard.
Incorrect positivity calls
When you encounter an unexpected or incorrect positivity call value, there are a few things to check:
Check that the Visit values are parsing correctly from the description field by looking at the following columns of the imported assay results: Specimen ID, Participant ID, Visit ID, Date, and Extra Specimen Info
Check that the run settings for the positivity calculation are as expected for the following fields: Calculate Positivity, Baseline Visit, and Positivity Fold Change
When the "Calculate Positivity" run property is set, the Analyte properties section will contain input fields for the "Positivity Thresholds" for each analyte. Check to make sure those values were entered correctly
Positivity calls for titrated unknowns will be made using only the data for the lowest dilution of the titration
Resources on PanoramaWeb - See tutorials, video demonstrations, and published experiments on PanoramaWeb.org.
Discovery Proteomics
Premium Resource - The ms2 module is a legacy feature set available to existing users of it with a Premium Edition of LabKey Server. Learn more or contact LabKey.
Managing, analyzing, and sharing high volumes of tandem mass spectrometry data, employing open-source tools provided by the Trans Proteomic Pipeline, developed by the Institute for Systems Biology
Searches against FASTA sequence databases using tools such as X! Tandem, Sequest, Mascot, or Comet
Analysis by PeptideProphet and ProteinProphet
Quantitation analysis on scored results using XPRESS and Q3
Documentation
You can find documentation of the ms2 module and accompanying legacy features in the documentation archives here:
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Neutralizing Antibody assays are designed to measure the effectiveness of therapeutic drugs and are often a critical part of demonstrating immune responses. They are particularly challenging to develop and validate due to the large volumes of diverse data generated.The NAb assay in our sample data is a plate-based assay that records neutralization in TZM-bl cells as a function of a reduction in Tat-induced luciferase (Luc) reporter gene expression after a single round of infection (Montefiori, D.C., 2004). See related resources below.The LabKey Server tools import the results from an Excel spreadsheet and provide management and analysis dashboards for the data. Both high- and low-throughput NAb assays are supported with options for cross- or single-plate dilutions, as well as an option for multiple viruses per plate. (For details, see NAb Plate File Formats.)Basic procedures for importing and working with assay data in general are covered in the Assay Tutorial. When working with a plate-based assay, there is an additional step of adding a plate template to the assay design, which is covered in the tutorial here and described in more detail in Specialty Plate-Based Assays.Dilution and well data for NAb assays is stored in the database in two tables, DilutionData and WellData. Users can write queries against these new tables, as well as export data from them.
Neutralizing Antibody (NAb) assays are plate-based and can consist of either high- or low-throughput formats with dilutions either across multiple plates or within a single plate. Further, multiple viruses and associated controls may be configured on a given plate template.This tutorial walks you through the process of creating a NAb assay design, including defining a plate template, then importing some sample data and reviewing options for working with it. Our sample data here came from a high-throughput 384 well plate with dilution across a single plate. When you input this high-throughput data, you have the option to bypass some data entry with an uploadable metadata file. If you are specifically interested in low-throughput NAb assays, you can also review the walkthrough in Work with Low-Throughput NAb Data.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
An assay design describes to LabKey Server how to interpret uploaded instrument data. For a NAb assay, that includes specifying what samples, controls, and viruses are in each well of the experimental plate. The sample data included with the tutorial matches a default template and design, but you can customize either or both to suit your own experiment. To begin the Tutorial: NAb Assay, you will first create a workspace, then create a plate template and assay design.
Download and unzip the sample data package LabKeyDemoFiles.zip. You will upload files from this unzipped [LabKeyDemoFiles] location later.
Log in to your server and navigate to your "Tutorials" project. Create it if necessary.
If you don't already have a server to work on where you can create projects, start here.
If you don't know how to create projects and folders, review this topic.
Create a new subfolder named "NAb Assay Tutorial". Choose the folder type "Assay."
Create a New NAb Plate Template
Assay designs may be created from scratch, or we can use pre-configured designs for specific assay types which are already customized with commonly used fields for the specific type of assay. In the case of a plate-based assay like NAb, first we create a plate template, which describes the contents in each well of the plate.
You should be in the "NAb Assay Tutorial" folder you created.
In the Assay List web part, click Manage Assays.
Click Configure Plate Templates.
Select new 384 well (16x24) NAb high-throughput (single plate dilution) template from the dropdown.
Click Create.
In the Plate Template Editor:
Enter Template Name: "NAb Tutorial Plate 1".
Make no other changes.
Click Save & Close.
You'll see your new template on the Available Plate Templates list.
This default template works with our sample data. When working with your own data and plates, you might need to customize the template as described in Customize NAb Plate Template.
Create a New NAb Assay Design
Next we create a new assay design which uses our new plate template. Our sample data is from a high-throughput NAb assay in which dilutions occur within a single plate. In addition, the instrument used here provides metadata about the experiment in its own file separate from the data file. For more about metadata input options, see NAb Plate File Formats.
Click the link NAb Assay Tutorial to get to the main folder page.
Click New Assay Design in the Assay List web part.
Click Specialty Assays.
Choose TZM-bl Neutralization (NAb), High-throughput (Single Plate Dilution) as your Instrument Specific Data Format.
Under Assay Location, select "Current Folder (NAb Assay Tutorial)".
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
When you import assay data, you declare how you will identify your data for later integration with other related data. See Data Identifiers for more details. In this case we'll use SpecimenIDs provided in the sample file, which match SpecimenIDs used in our LabKey demo study.
Import NAb Data
Locate the LabKeyDemoFiles package you downloaded and unzipped in the previous step. The two files you will upload in this step are in the [LabKeyDemoFiles]/Assays/NAb/ directory.
Return to the NAb Assay Tutorial page if you navigated away.
In the Assay List web part, click NAb Tutorial Assay Design.
Click Import Data.
For Participant/Visit (i.e. how you will identify your data):
Select Specimen/sample id.
Do not check the box for providing participantID and visitID.
You do not need to select a target study at this time.
Click Next.
On the data import page:
Leave the Assay ID blank. The name of the data file will be used as the AssayID by default.
For Cutoff Percentage (1) enter 50.
From the Curve Fit Method pulldown, select Five Parameter.
For Sample Metadata:
Click Choose File (or Browse).
Select "NAb_highthroughput_metadata.xlsx" from the [LabKeyDemoFiles]/Assays/NAb/ directory.
For Run Data select "NAb_highthroughput_testdata.xlsx" from the same sample location.
When the import is complete, the run summary dashboard gives you a quick way to validate the data. In the next step we will review in more detail the information and options available here.
Troubleshooting
If you see a message like this:
There was an error parsing NAb_highthroughput_testdata.xlsx. Your data file may not be in csv format.
It indicates that your plate template was created using the wrong format. You may have selected the 96 well variant instead of the 384 well one. Return to the previous step, create a new plate template of the correct format with a new name, then edit your assay design to select it instead and repeat the import.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
High-throughput 384-well NAb assays may contain hundreds of samples with dilutions across plates or within a single plate and the resulting graphs and views can be complex. The LabKey NAb Assay tools provide quick visual feedback allowing you to confirm a valid run or immediately correct and rerun if necessary.
Review NAb Dashboard
After uploading a NAb run, you will see the NAb Dashboard. The Run Summary section includes the percent neutralization for each dilution or concentration, calculated after subtraction of background activity. The NAb tool fits a curve to the neutralization profile using the method you specified when uploading the run (in this tutorial example a five-parameter fit). It uses this curve to calculate neutralizing antibody titers and other measures. The tool also calculates “point-based” titers by linearly interpolating between the two replicates on either side of the target neutralization percentage.The percent coefficient of variation (%CV) is shown on the neutralization curve charts as vertical lines from each data point.If you are not working through the tutorial on your own server, you can view a similar dashboard in the interactive example.If you navigate away from the dashboard you can return to from the Run view by clicking the info icon .Below the graphs, a data summary by specimen and participant includes:
AUC -- Area Under the Curve. This is the total area under the curve based on the titrations, with negative regions counting against positive regions.
PositiveAUC -- Positive Area Under the Curve. This figure represents only the areas under the curve that are above the y-axis.
Even lower on the page, you'll find even more detailed specimen and plate data.
Quality Control
An administrator can review and mark specific wells for exclusion from calculations. See NAb Assay QC for details.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
In this step, we tour some of the graph options available for Neutralizing Antibody (NAb) Assay data.
Explore Graph Options
The Change Graph Options menu at the top of the run details page offers a variety of viewing options for your data:
Curve Type: See what the data would look like if you had chosen a different curve fit (see below).
Graph Size: Small, medium, or large graphs as desired.
Samples Per Graph: Choose more graphs each containing fewer samples, or fewer more complex graphs. Options: 5, 10, 15, 20 samples per graph.
Graphs Per Row: Control the width of the results layout. Options 1, 2, 3, or 4 graphs per row.
Data Identifiers: If you provided multiple ways of identifying your data, you may select among them here.
From the Change Graph Options menu, select a different Curve Type and view the resulting graph.
You can experiment with how the graph would appear using a different curve fit method without changing the run data. For example, select Polynomial. The top graph will look something like this:
Note that this alternate view shows you your data with another curve type selected, but does not save this alternate view. If you want to replace the current run data with the displayed data, you must delete and reimport the run with the different Curve Type setting.
Graph URL Parameters
Notice that as you change graph options, the page URL is updated with the parameters you changed. You can customize the graph directly via the URL if you wish. In fact, while the 96-well low-throughput NAb assays do not offer all of these additional graph options on the pulldown menus, if you would like to use them you could specify the same parameters in the URL. For example:
LabKey's NAb Assay tools provide quick feedback after the upload of each run. Once you confirm that the particular run of data is valid you might want to quickly share the results with colleagues via URLs or printable browser view. You could also copy your data to a LabKey study where it could be integrated with other information about the same specimens or samples. Connecting differing neutralization rates to different cohorts or treatment protocols could enable discoveries that improve results.You have now completed the process of setting up for, importing, and working with NAb data. You might proceed directly to designing the plate template and assay that would suit your own experimental results. If you are working with low-throughput 96-well data, you can learn more in this topic: Work with Low-Throughput NAb Data.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
The NAb Assay Tutorial covers the process of working with NAb data using a high-throughput assay and uploading data and metadata from files. This topic covers importing data from a low-throughput NAb assay. This process requires more data entry of specific metadata, and gives you the option to see the use of multiple data identifiers in action.
Select new 96 well (8x12) NAb single-plate template from the dropdown.
Click Create to open the editor.
Specify the following in the plate template editor:
Provide a Template Name, for example: "NAb Plate 1"
Leave all other settings at their default values (in order to create the default NAb plate).
Click Save & Close.
Select > Manage Assays.
Click New Assay Design.
Click the Specialty Assays tab.
Under Use Instrument Specific Data Format, select TZM-bl Neutralization (NAb).
As the Assay Location, select "Current Folder (NAb Assay Tutorial)" (your folder name may vary).
Click Choose TZM-bl Neutralization (NAb) Assay.
Specify the following in the assay designer:
Name: "NAbAssayDesign"
Plate Template: NAb Plate 1
Metadata Input Format: Confirm Manual is selected for this tutorial.
Leave all other fields at their default values.
Click Save.
Import Data
When importing runs for this tutorial, you provide metadata, in this case sample information, manually through the UI. If instead you had a file containing that sample information, you could upload it using the File Upload option for Metadata Input Format. See NAb Assay Reference for more information.
Select > Manage Assays.
On the Assay List, click NAbAssayDesign, then Import Data.
For Participant/Visit, select Specimen/sample id. Do not check the box to also provide participant/visit information.
Click Next and enter experiment data as follows:
Property
Value
Assay Id
Leave blank. This field defaults to the name of the data file import.
Cutoff Percentage (1)
50
Cutoff Percentage (2)
80
Host Cell
T
Experiment Performer
<your name>
Experiment ID
NAb32
Incubation Time
30
Plate Number
1
Curve Fit Method
Five parameter
Virus Name
HIV-1
Virus ID
P392
Run Data
Browse to the file: [LabKeyDemoFiles]\Assays\NAb\NAbresults1.xls
Specimen IDs
Enter the following (in the row of fields below the run data file):
Specimen 1
526455390.2504.346
Specimen 2
249325717.2404.493
Specimen 3
249320619.2604.640
Specimen 4
249328595.2604.530
Specimen 5
526455350.4404.456
Initial Dilution
Place a checkmark and enter 20
Dilution Factor
Place a checkmark and enter 3
Method
Place a checkmark and select Dilution
Click Save and Finish.
NAb Dashboard
When the import is complete, you can view detailed information about any given run right away, giving quick confirmation of a good set of data or identifying any potential issues. The run summary dashboard looks something like this:The percent coefficient of variation (%CV) is shown on the neutralization curve charts as vertical lines from each data point. Additional quality control, including excluding particular wells from calculations is available for NAb assays. See NAb Assay QC for details.As with high-throughput NAb data you can customize graphs and views before integrating or sharing your results. Once a good set of data is confirmed, it could be linked to a study or data repository for further analysis and integration.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Data Identifiers are used to define how assay data will be mapped to samples, specimens, participants, or other types of data. For NAb data, you can upload multiple ways to identify data at once and switch between them using graph parameters.
Data Identifiers
When you upload a NAb run and enter batch properties, you declare how you will identify the data by selecting an identifier known as a Participant/Visit Resolver. Choices include:
Participant id and visit id. If VisitID is not specified, it is set to null.
Participant id and date.
Participant id, visit id, and date. If VisitID is not specified, it is set to null.
Specimen/sample id. If you choose this option, you may also provide participant id and visit id.
Sample indices, which map to values in a different data source. This option allows you to assign a mapping from your own specimen numbers to participants and visits. The mapping may be provided by pasting data from a tab-separated values (TSV) file, or by selecting an existing list. Either method must include an 'Index' column and using the values of the columns 'SpecimenID', 'ParticipantID', and 'VisitID'. To use the template available from the Download template link, fill in the values, copy and paste the entire spreadsheet including column headers into the text area provided.
For example, if you choose Specimen/sampleID, as we did in this tutorial, the specimenID field will be used to identify the data. If you had also checked the box to supply Participant/visit identifiers, you would have the option to select that option from the NAb Details page using Change Graph Options > Data Identifiers.Options on this menu will only be enabled when there is enough data provided to use them. The tutorial example does not include providing this second set of identifiers, but you may try this feature yourself with low-throughput NAb data. Note that the data identifier selection is included as a URL parameter so that you may share data with or without this graph option.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Ensuring the quality and reliability of NAb assay results is made easier with a set of quality control (QC) options built in to the assay tools. Removing ill-fitted and otherwise unsuitable data within LabKey saves users performing these tasks using outside tools. To review and mark data for exclusion, the user must have administrator access. Other users can see the QC report once created.
Open the details view of the run you want to review. From the Assay List, click the assay name, then Run Details for the desired row.
Select View QC > Review/QC Data. If you do not see this menu option, you do not have permission to perform this step.
The review page shows the run result data, with checkboxes for each item.
QC Review Page
The page is divided into sections which may vary based on the type and complexity of data represented. In this example, a single low-throughput plate containing 5 specimens at varying dilutions is represented with a section for the plate controls, followed by a series of graphs and specific data, one set for each specimen.
Check the box(es) for any data you would like to exclude.
Scroll to the bottom of the page and click Next.
The QC summary page allows you to enter a comment for each exclusion and shows the plate with excluded wells highlighted:
If you notice other data you would like to exclude, you can click Previous and return to the selection page to add or delete checkmarks. When you return to the summary by clicking Next, any previously entered comments have been preserved.
Click Finish when finished to save the exclusions and recalculate results and curve fits.
View Excluded Data
After some data has been excluded, users with access to view the run details page will be able to tell at a glance that it has been reviewed for quality control by noticing the Last Reviewed for QC notation in the page header.
On the run details page, the user and date are now shown under Last Reviewed for QC and excluded wells are highlighted in red.
Hovering over an excluded well will show a tooltip containing the exclusion comment. If none was entered the tooltip will read: "excluded from calculations".
Users can review the QC report by selecting View QC > View Excluded Data from the run details page. The "Excluded Data" report looks like the QC report summary page above, including the comments entered.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Neutralizing Antibody (NAb) assays can be configured so that multiple viruses are tested on a single plate. The LabKey NAb assay design can then interpret these multiple virus and control well groups on that plate, so that results of each run may be viewed and graphed on a per-virus basis.
Configure a Multi-Virus Plate Template
The built-in NAb Multi-virus Plate Template is divided in half between two viruses with 10 samples (in replicate) per virus, each with their own separate control wells. The samples on each plate are identical between the two viruses. By customizing this built-in default plate template, it is further possible to add additional viruses or otherwise customize the template to fit alternate arrangements of well groups and plates.
Select > Manage Assays.
Click Configure Plate Templates.
Select new 384 well (16x24) NAb multi-virus plate template from the dropdown.
Click Create to open the editor.
Notice the Virus tab which will show you the default layout of the two viruses.
After making any changes necessary, name your template.
Click Save & Close.
Define a Multi-Virus Single-Plate Assay Design
Select the appropriate NAb assay type as you create a new named assay design.
Select > Manage Assays.
Click New Assay Design.
Click the Specialty Assays tab.
Select "TZM-Bl Neutralization (NAb)" as the assay type and the current folder as the location.
Click Choose TZM-Bl Neutralization (NAb) Assay.
Name the design.
As Plate Template, select the multi-virus template you just created and named above (it may be selected by default).
Open the section for Virus Fields to see the two included by default: VirusName and VirusID. These will be included with other properties populated during import.
Add additional fields as needed.
Click Save.
Import Multi-Virus NAb Data
During upload of data, the upload wizard will request the virus specific information and other metadata necessary to correctly associate data from each well with the correct virus. Dilution and neutralization information is also grouped by virus.
Explore Multi-Virus Results
In the resulting run details report, each sample/virus combination will have its own set of dilution curves, cutoffs, AUC, fit errors, etc.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Neutralizing Antibody (NAb) Assays can be of several different types:
Low-throughput assays typically have 96 wells in a single plate, prepared with five specimens in eight dilutions of two replicates each. Low-throughput samples are diluted within a single plate.
High-throughput assays typically have 384 wells per plate and may consist of up to eight plates. High-throughput assays have two options for dilution:
Cross Plate Dilution: Each well on a given plate has the same dilution level; dilutions occur across plates.
Single Plate Dilution: Dilutions occur within a single plate.
Multi-Virus NAb assays include multiple viruses within a single plate and may be either low- or high-throughput and either single- or cross-plate dilution.
The specific file format generated by your plate reader will determine how you configure your assay design to properly parse the data and metadata it contains.
Low-Throughput NAb Assay Formats
LabKey's low-throughput NAb assay supports a few different formats. Files containing these formats may be of any type that the TabLoader can parse: i.e. Excel, tsv, csv, txt.
format1.xls has the plate data in a specific location on the second sheet of the workbook. The 96 wells of plate data must be in exactly the same location every time, spanning cells A7-L14.
SpectraMax.csv contains plate data identified by a "Plate:" cell header.
format2.xls follows a more general format. It can have the plate data on any sheet of the workbook, but the rows and columns must be labeled with 1-12 and A-H.
format3.tsv is the most general format. It is a file that just contains the plate data without row or column headers.
For all formats, only the plate data is read from the file. All other content, including worksheet naming, is ignored.
Metadata Input Format
For low-throughput NAb assays, sample and virus metadata input may be done manually via form input at assay import time, or may be uploaded as a separate file to simplify the import process and reduce user error. For example, if you are using a FileMaker database you can export the information and upload it directly. The file upload option supports Excel, tsv, csv, or txt file types.
In order to support different plate readers for high-throughput 384-well NAb assays, LabKey tools support two methods of uploading metadata. Some instruments output metadata and run data in separate spreadsheets, others generate a combined file which contains both. As part of new LabKey Assay design, you select a Metadata Input Format of either:
File Upload (metadata only): upload the metadata in a separate file.
Combined File Upload (metadata & run data): upload both in a single file.
If you are importing a run where the metadata is uploaded in a separate file, you can download a template from the run properties page by clicking Download Template.
Multi-Virus NAb Assay Format
When working with multiple viruses on a single plate, the assay design can be configured to specify multiple viruses and control well groups within a single plate design so that a run may be uploaded at once but results viewed and graphed on a per-virus basis. See Work with Multiple Viruses per Plate for details.
Customize NAb Plate Template
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
The NAb Assay design tools offer a range of default plate template layouts, which you may further customize to create a template for your assay design that matches the exact layout of the controls, viruses, samples, specimens, and replicates on your plate or plates.
The default, low-throughput NAb plate template is called "NAb: 5 specimens in duplicate" and corresponds to the following plate layout:This plate tests five samples in duplicate for inhibition of infection and thus decreased luminescence. The first column of eight wells provides the background signal, measured from the luminescence of cells alone, the “Virus Control.” The second column of eight wells, the “Cell Control” column, provides the maximum possible signal. This is measured from cells treated with the virus without any antibody sample present to inhibit infection. These two columns define the expected range of luminescence signals in experimental treatments. The next five pairs of columns are five experimental treatments, where each treatment contains a serial dilution of the sample.
Create a NAb Assay Plate Template
Select > Manage Assays.
Click Configure Plate Templates.
The plate templates already defined (if any) are listed with options to:
Edit an existing template.
Edit a Copy creating a new variant of an existing template, but leaving the original unchanged.
Copy to Another Folder to create a new variant in another location.
Delete.
To start a new template, choose one of the built-in NAb templates from the dropdown.
Your choice depends on the number of wells, layout, and dilution method you used for your experiments.
Click Create to open the plate editor. Examples are below.
Name the template. Even if you make no changes, you need to name the base template to create a usable instance of it.
Once you have created a template, you will see it available as a dropdown option when you create assay designs and import data.
Customize a NAb Assay Plate Template
Customize an assay plate template to match the specific plate and well layout used by your instrument.
Open the plate editor as described above.
Explore the Control, Specimen, Replicate, and Other tabs.
Edit as desired on any of the tabs.
On the Control tab, the well groups must always be named "VIRUS_CONTROL_SAMPLE" and "CELL_CONTROL_SAMPLE" and you cannot add new groups. You can click or drag to apply these groups to cells.
On the other tabs, you will see existing/built-in groups (if any) and be able to add other named groups if needed. For instance, on the Specimen tab, you can add the necessary number of specimen groups by entering the new group name and clicking Create. Then select a specimen (using the color coded radio buttons under the layout) and either click individual cells or drag across the plate template editor to "paint" showing where the chosen specimen will be located on the plate. There are also buttons to shift the entire array Up, Down, Left, or Right.
Warnings, if any, will be shown as well. For example, if you identify a given well as both a specimen and control group, a warning will be raised.
Click Save and Close.
Reverse Dilution Direction
Single-plate NAb assays assume that specimens get more dilute as you move up or left across the plate. High-throughput NAb assays assume that specimens are more dilute as you move down or right across the plate. To reverse the default dilution direction for a specimen well group, select it, add a well group property named 'ReverseDilutionDirection', then give it the value 'true.'
Example Plate Template Editor Layouts
The plate template editor for low-throughput NAb assays, showing the Control tab with the two built-in group names, "CELL_CONTROL_SAMPLE" and "VIRUS_CONTROL_SAMPLE" that are required by the system:The Virus tab lets you define your own New well groups by typing a name and clicking Create, or Create Multiple to generate a numbered series.The plate template editor for a 384-well high-throughput NAb assay with cross plate dilution, showing the specimen layout:
Default NAB assay designs include properties beyond the default properties included in Standard assay designs. For any TZM-bl Neutralization (NAb) assays, the following additional properties can be set.
Assay Properties
Plate Template: The template that describes the way your plate reader outputs data. You can:
Choose an existing template from the drop-down list.
Metadata Input Format: Assays that support more than one method of adding sample/virus metadata during the data import process can be configured by selecting among possible options. Not all options are available for all configurations:
Manual: Metadata is entered manually at the time of data import. Available for low-throughput NAb assays only.
File Upload (metadata only): Metadata is uploaded in a separate file from the actual run data at the time of data import.
Combined File Upload (metadata & run data): Upload a combined file containing both metadata and run data. Available only for some high-throughput NAb assays.
Lock Graph Y-Axis (True/False): Fixes the Y axis from -20% to 120%, useful for generating graphs that can easily be compared side-by-side. If not set, axes are set to fit the data, so may vary between graphs.
Curve Fit Method. Required. The assay's run report (accessed through the details link) generates all graph, IC50, IC80 and AUC information using the selected curve fit method. You can choose from the following types of curve fits:
Five parameter (5-pl): A generalized version of the Hill equation is used.
Four parameter (4-pl): A generalized version of the Hill equation is used.
Polynomial: This algorithm allows you to quantifying a sample’s neutralization behavior based on the area under a calculated neutralization curve, commonly abbreviated as “AUC”.
Sample Fields
For each run, the user will be prompted to enter a set of properties for each of the sample well groups in their chosen plate template. In addition to Standard assay data property fields, which include date and participant/visit resolver information, NAb assays include:
Sample Description. Optional.
Initial Dilution. Required. Sample value: 20.0. Used for calculation.
Dilution Factor. Required. Sample value: 3.0. Used for calculation.
Method. Required. Dilution or Concentration.
For more information and a tutorial on designing and using NAb Assay Tools, see Tutorial: NAb Assay.
View Additional Data
The default views of run and result data do not display all the available information. You can also find the following information:
The NAb assay results grid has a hidden lookup FK column to allow for the existing per-sample dilution data to be pulled into the results grid view.
Select > Manage Assays.
Click your assay name to open the runs page.
Click View Results.
Select (Grid View) > Customize Grid.
Check the box to Show hidden fields (at the bottom of the available fields panel).
Scroll to find the Dilution Data node and click to expand it.
Check boxes for the data you want to show in your results grid.
Per-run Cell and Virus Control Aggregate Data
New aggregate queries were added to the NAb assay schema to view per-run cell and virus control aggregate data. For each run, you can see the following for each control:
Avg Value
Min Value
Max Value
Std Dev Value
Select > Go To Module > Query.
Open the assay schema, then the folder for your NAb Assay type (such as "Single Plate Dilution NAb" and) and assay design.
Under built-in queries & tables click the name of the table you want to view.
CellControlAggregates
VirusControlAggregates
Click View Data to see the data.
To pull these values into the Runs grid view:
Select > Manage Assays.
Click your assay name to open the runs page.
Select (Grid View) > Customize Grid.
Check the box to Show hidden fields (at the bottom of the available fields panel).
Scroll to find the Cell Control Aggregates and/or Virus Control Aggregates nodes.
Expand the node and check boxes for the data you want to show in your results grid.
Show Curve Fit Parameters
A new "Fit Parameters" field was added to the Nab assay results grid to store the curve fit parameters based on the selected curve fit method for a given run. This new field was populated via an upgrade script for all existing NAb runs.
Select > Manage Assays.
Click your assay name to open the runs page.
Click View Results.
Select (Grid View) > Customize Grid.
Check the box for Fit Parameters and click View Grid to see them in the grid (you can drag them to anywhere in the column list you like).
The Fit Parameters are shown as a set of key value pairs helping you identify the curve fit method used.
Premium Feature — Available upon request with all Premium Editions of LabKey Server. Learn more or contact LabKey.
The assay request module expands the functionality of LabKey's generic issue tracker, providing a workflow designed especially for the assaying of specimens and samples, providing a collaboration tool that ties together the various elements of the work to be completed in a single trackable ticket. The module ties together the following elements:
the sample(s) to be processed
the kind of assay to be run (NAb, Luminex, ELISA, etc.)
the lab technician responsible for completing the assay processing
the original requester who will sign off on the completed work
Overview
The request dashboard shows at a glance the number and status of assay requests.When a lab technician is running the requested assay, they select which request they are fulfilling to assist in connecting this information.
Premium Resources AvailableSubscribers to premium editions of LabKey Server can learn more about how to configure and use the Assay Request Tracker in these topics:
LabKey Server's Experiment module provides a framework for describing experimental procedures and for transferring experiment data into and out of a LabKey Server system. An experiment is a series of steps that are performed on specific inputs and produce specific outputs.
Experiments can be described, archived and transferred in experiment descriptor files (XAR files). For example, Assay Designs can be saved in a XAR file format.
The basic terms and concepts in the LabKey Server experiment framework are taken from the Functional Genomics Experiment (FuGE) project. This covers a subset of the FuGE object model. More details on FuGE can be found at http://fuge.sourceforge.net.
The LabKey Server experiment framework uses the following primary objects to describe an experiment.
Sample or Material: These terms are synonyms. A Sample object refers to some biological sample or processed derivative of a sample. Examples of Sample objects include blood, tissue, protein solutions, dyed protein solutions, and the content of wells on a plate. Samples have a finite amount and usually a finite life span, which often makes it important to track measurement amounts and storage conditions for these objects. Samples can be included in the description of an experiment as the input to a run. The derivation of Samples can be tracked.
Sample Type A Sample Type is a group of Samples accompanied by a suite of properties that describe shared characteristics of all samples in the group.
Data: A Data object refers to a measurement value or control value, or a set of such values. Data objects can be references to data stored in files or in database tables, or they can be complete in themselves. Data objects can be copied and reused a limitless number of times. Data objects are often generated by instruments or computers, which may make it important to keep track of machine models and software versions in the applications that create Data objects.
Protocol or Assay: These terms are synonyms. A Protocol object is a description of how an experimental step is performed. A Protocol object describes an operation that takes as input some Sample and/or Data objects, and produces as output some Sample and/or Data objects. In LabKey Server, Protocols are nested one level--an experiment run is associated with a parent protocol. A parent protocol contains n child protocols which are action steps within the run. Each child protocol has an ActionSequence number, which is an increasing but otherwise arbitrary integer that identifies the step within the run. Child protocols also have one or more predecessors, such that the outputs of a predecessor are the inputs to the protocol. Specifying the predecessors separately from the sequence allows for protocol steps that branch in and out. Protocols also may have ParameterDeclarations, which are intended to be control settings that may need to be set and recorded when the protocol is run.
ProtocolApplication: The ProtocolApplication object is the application of a protocol to some specific set of inputs, producing some outputs. A ProtocolApplication is like an instance of the protocol. A ProtocolApplication belongs to an ExperimentRun, whereas Protocol objects themselves are often shared across runs. When the same protocol is applied to multiple inputs in parallel, the experiment run will contain multiple ProtocolApplications object for that Protocol object. ProtocolApplications have associated Parameter values for the parameters declared by the Protocol.
ExperimentRun: The ExperimentRun object is a unit of experimental work that starts with some set of input materials or data files, executes a defined sequence of ProtocolApplications, and produces some set of outputs. The ExperimentRun is the unit by which experimental results can be loaded, viewed in text or graphical form, deleted, and exported. The boundaries of an ExperimentRun are up to the user.
RunGroup or Experiment: These terms are synonyms. LabKey Server's user interface calls these entities RunGroups while XAR.xml files call them Experiments. A RunGroup is a grouping of ExperimentRuns for the purpose of comparison or export. The relationship between ExperimentRuns and RunGroups is many-to-many. A RunGroup can have many ExperimentRuns and a single ExperimentRun can belong to many RunGroups.
XAR file: A compressed, single-file package of experimental data and descriptions. A XAR file expands into a single root folder with any combination of subfolders containing experimental data and settings files. The archive includes a xar.xml file that serves as a manifest for the contents of the XAR as well as a structured description of the experiment that produced the data.
Relationships Between Experiment Objects
At the core of the data relationships between objects is the cycle of ProtocolApplications and their inputs and outputs, which altogether constitute an ExperimentRun.
The cycle starts with either Sample and/or Data inputs. Examples are a tissue sample or a raw data file output from an LCMS machine.
The starting inputs are acted on by some ProtocolApplication, an instance of a specific Protocol that is a ProtocolAction step within the overall run. The inputs, parameters, and outputs of the ProtocolApplication are all specific to the instance. One ProtocolAction step may be associated with multiple ProtocolApplications within the run, corresponding to running the same experimental procedure on different inputs or applying different parameter values.
The ProtocolApplication produces sample and/or data outputs. These outputs are usually inputs into the next ProtocolAction step in the ExperimentRun, so the cycle continues. Note that a Data or Sample object can be input to multiple ProtocolApplications, but a Data or Sample object can only be output by at most one ProtocolApplication.
The relationships between objects are intrinsically expressed in the relationships between tables in the LabKey Server database as shown in the following diagram:
The Experiment module provides a framework for describing experimental procedures and for transferring experiment data into and out of a LabKey Server system. An experiment is a series of steps that are performed on specific inputs and produce specific outputs.
Experiment runs can be accessed and managed through the Experiment Runs web part. The runs available in the current folder are listed, by default starting with "All Runs" and then listing more specific run types below.
Graph Links: Click to go directly to the lineage graph for this run.
Add to Run Group: Select one or more rows, then add to a run group. You can create a new run group from this menu.
Move: Move the run to another container that has been configured with a pipeline root. The possible destinations will be shown as links.
Create Run: This button is available when the folder includes assay designs or other run building protocols. The list of options for run creation is available on the menu.
You can also access this view by selecting > Go To Module > Experiment.
Customize the Experiment Runs Web Part
When you are viewing the Experiment Runs web part, filtering is provided on the (triangle) > Customize menu. The default in most folders is to show all runs. If you access > Go To Module > Experiment directly, you can use Filter by run type to select how to display runs.By selecting <Automatically selected based on runs> you will see the runs grouped by type.
Create Run
Creation of a new run is available for some protocols, including assay designs.Select Create Run in the experiment runs menu bar. The menu will list assay designs and other experiment protocols that are defined in the folder. Select the option you want to use to create the run.Creating a run for an assay design will open the assay data importer on the batch properties panel.
Premium Features AvailableWith a Premium Edition of LabKey Server and an additional module, you can enable an interactive run builder for creating experiment runs. Learn more in this topic:
Run groups allow you to assign various types of runs (experiment runs, assay runs, etc) to different groups. You can define any groups that you like. For example, separate groups for case and control, a group to hold all of your QC runs, or separate groups for each of the different instruments you use in the lab. Run groups are scoped to a particular folder.
Create Run Groups and Associate Runs with Run Groups
From a list of runs, select the runs you want to add to the group and click Add to run group. Select from existing groups (if any) or choose Create new run group.To add Standard assay runs to a run group, use the Experiment Runs web part.When you define a new group, you must give it a name, and can provide additional information if you like. Click Submit to create the run group and add the runs you selected to it.Continue this process to define all the groups that you want.The "Run Groups" column will show all of the groups to which a run belongs.
Viewing Run Groups
You can click the name of one of the groups in the Run Groups column of a run list to see details and member runs. You can also add the Run Groups web part, or access it through the Experiment module ( > Go to Module > Experiment).You can edit the run group's information, as well as view all of the run group's runs. LabKey Server will attempt to determine the most specific type of run that describes all of the runs in the list and give you the related set of options.
Viewing Group Information from an Individual Run
From either the text or graphical view of an experiment run, you have access to a list of all the run groups in the current folder.
Filtering a Run List by Run Group Membership
You can add columns to your list of runs that let you filter by run group membership. In the MS2 Runs web part, for example, select (Grid View) > Customize Grid. Expand the Run Group Toggle node. Check the boxes for the group or groups that you want to add (in this example, we choose both "K Score" and "Native Score"). Click Save.Your run list will now include columns with checkboxes that show if a run belongs to the group. You can toggle the checkboxes to change the group memberships. You can also add a filter where the value is equal to TRUE or FALSE to restrict the list of runs based on group membership.
Experiment Lineage Graphs
Lineage relationships established through the experiment framework or using the run builder are displayed on the Run Details panel. Either start from the Run Details and click the Graph Summary View tab, or click the graph icon in the Experiment Runs web part.
The Graph Summary View shows the overall run process. The currently 'selected' node will show Details to the right and you can click to select other nodes for their details.You may instead see a legacy graph format and a Toggle Beta Graph button to switch to the above view. Learn more below).
Run Properties
The Run Properties tab on the Graph Summary View presents the properties defined for this run.
Graph Detail View
The Graph Detail View tab focuses on one part of the graph. Click any node to see the detail page.
Text View
The Text View tab gives a table and text based overview of the run.
Legacy Experiment Run Graph Notation
Experiment run graphs are presented using the following graphical notation representing the flow of the experiment.For example, the above graph is represented like this in the legacy view:
LabKey Server's provenance module is available with Premium Editions of LabKey Server, but is not currently included in all distributions. Please contact LabKey to inquire about obtaining and using the Run Builder feature described here.
The run builder captures experiment processes by linking together 1) a set of inputs, 2) a process or "run", and 3) a set of outputs. This user-friendly tool prompts the user for information about the run: the inputs (samples and/or files), metadata on the run that was performed, and the output (samples and/or files). Any combination of samples and files can be linked to a run, either as inputs or outputs, making this a flexible tool that can be applied to many use cases.The run builder can be useful for capturing "at the bench" experimental information. For example, a lab technician might use the run builder to record that a process has happened, where samples were the input and files were the output. The run builder lets you form relationships between individual input samples/files (or any other LSID record) with individual outputs. In the run depicted below, a blood sample is the input to the experiment and two CSV files form the output:Samples and files may be selected from those already loaded onto the server, or created during the run building process. Run inputs and outputs create lineage/parentage relationships, which are reflected in the lineage views as parents (inputs) and children (outputs).Once they are related within a run, the files and samples cannot be deleted without first deleting the run.Note that the "runs" referred to here are experiment runs, not be confused with Assay runs.
Enable Run Builder
To enable the run builder, you must enable both the provenance and experiment modules within the container where you plan to use it.
Select > Folder > Management.
Click the Folder Type tab.
Check the boxes for Experiment and Provenance.
Click Update Folder.
Create Runs
You can use the Run Builder to create runs from within the experiment module, from a sample grid, from the file browsers, or directly using the URL.
Access Run Builder from Experiment Runs
You can access the run builder from the Experiment Runs menu bar. If your folder does not already include this web part, add "Experiment Runs" to your page, or select > Go To Module > Experiment.
Viewing the list of samples in a Sample Type, you can also select the desired samples and use Create Run > Input Samples or Output Samples depending on how you want the samples included in the run you build:When the run builder opens, it will be prepopulated with the samples you selected, in either the Inputs or Outputs section. You can add additional samples and files before saving the run.
Create Runs from Files
When the provenance module is enabled, the file browser includes a Create Run button. Access it via the Files web part or > Go To Module > FileContent.Select the file of interest using the checkbox and click Create Run. In the popup, select either Inputs or Outputs depending on which section should include the chosen file(s). Click Create Run.The run builder will open with the chosen files prepopulated in the selected section. You can add additional files and samples to the run before saving.
Access Run Builder via URL
The run builder can also be accessed directly via the URL. Navigate to the desired container and replace the end of the URL with:
provenance-runBuilder.view?
Define Properties
In the Properties section, give your run a name, enter a description, and add any Vocabulary Properties with values that you need. These values will be applied to the run as a whole, not to individual inputs or outputs.Under Inputs and Outputs, add the samples and files required.You can Save while building the run, and click Save & Finish when finished.
Add Samples
To add samples, either to the Inputs or Outputs section of the run builder, click the Add Samples button.You can add samples in two ways:
Use the Copy/Paste Samples tab to select samples by pasting a list of sample names into the box. Names can be separated by tabs, newlines, commas, or spaces. Clicking Add # Samples will add the samples to the run.
Select Find Samples to choose samples from a grid, as shown below:
Select the sample type and the selection panel and grid below will populate.You can filter, sort, or search to locate the samples you want to include. Click in the box to use it to locate your samples of interest.Check the boxes for the desired samples and click Add Samples. The samples you added will now appear in a grid in the run builder:You can now click Add Samples to add more, Move Selected Samples to Output or if necessary, select and Remove Selected Samples.You can Save work in progress while building the run, and click Save & Finish when finished.
Add Files
Click Add Files to choose files to add to either the input or output sections. Use the File Select tab to browse the file root for files already uploaded. Use the checkboxes to select one or more files to add to the run you are building.Click the File Upload tab to switch to a drag-and-drop interface for uploading new files.Click Add Files when ready.The files you've added will be listed, each with a for deleting. Click Add Files to add more as needed.You can Save work in progress while building the run, and click Save & Finish when finished.
Experiment Runs Web Part
Runs created by the run builder are available on the Experiment Runs web part. Click the name to see run details; click the graph icon to the left of the run name to go directly to the lineage graph for this run. Learn more in this topic:
To edit a run, you can reopen the Run Builder interface by clicking Edit Run from the run details page.Make the necessary changes and click Save & Finish.
The LabKey Server platform uses the LSID standard for identifying entities in the database, such as experiment and protocol definitions. LSIDs are a specific form of URN (Universal Resource Name). Entities in the database will have an associated LSID field that contains a unique name to identify the entity.
Constructing LSIDs
LSIDs are multi-part strings with the parts separated by colons. They are of the form:urn:lsid:<AuthorityID>:<NamespaceID>:<ObjectID>:<RevisionID>The variable portions of the LSID are set as follows:
<AuthorityID>: An Internet domain name
<NamespaceID>: A namespace identifier, unique within the authority
<ObjectID>: An object identifier, unique within the namespace
<RevisionID>: An optional version string
An example LSID might look like the following:urn:lsid:genologics.com:Experiment.pub1:Project.77.3LSIDs are a solution to a difficult problem: how to identify entities unambiguously across multiple systems. While LSIDs tend to be long strings, they are generally easier to use than other approaches to the identifier problem, such as large random numbers or Globally Unique IDs (GUIDs). LSIDs are easier to use because they are readable by humans, and because the LSID parts can be used to encode information about the object being identified.Note: Since LSIDs are a form of URN, they should adhere to the character set restrictions for URNs. LabKey Server complies with these restrictions by URL encoding the parts of an LSID prior to storing it in the database. This means that most characters other than letters, numbers and the underscore character are converted to their hex code format. For example, a forward slash "/" becomes "%2F" in an LSID. For this reason it is best to avoid these characters in LSIDs.The LabKey Server system both generates LSIDs and accepts LSID-identified data from other systems. When LSIDs are generated by other systems, LabKey Server makes no assumptions about the format of the LSID parts. External LSIDs are treated as an opaque identifier to store and retrieve information about a specific object. LabKey Server does, however, have specific uses for the sub-parts of LSIDs that are created on the LabKey Server system during experiment load.Once issued, LSIDs are intended to be permanent. The LabKey Server system adheres to this rule by creating LSIDs only on insert of new object records. There is no function in LabKey Server for updating LSIDs once created. LabKey Server does, however, allow deletion of objects and their LSIDs.AuthorityIDThe Authority portion of an LSID is akin to the "issuer" of the LSID. In LabKey Server, the default authority for LSIDs created by the LabKey Server system is set to labkey.com.Note: According to the LSID specification, an Authority is responsible for responding to metadata queries about an LSID. To do this, an Authority would implement an LSID resolution service, of which there are three variations. The LabKey Server system does not currently implement a resolution service, though the design of LabKey Server is intended to make it straightforward to build such a service in the future.NamespaceIDThe Namespace portion of an LSID specifies the context in which a particular ObjectID is unique. Its uses are specific to the authority. LSIDs generated by the LabKey Server system use this portion of the LSID to designate the base object type referred to by the LSID (for example, Material or Protocol.) LabKey LSIDs also usually append a second namespace term (a suffix) that is used to ensure uniqueness when the same object might be loaded multiple times on the same LabKey Server system. Protocol descriptions, for example, often have a folder scope LSID that includes a namespace suffix with a number that is unique to the folder in which the protocol is loaded.ObjectIDThe ObjectID part of an LSID is the portion that most closely corresponds to the "name" of the object. This portion of the LSID is entirely up to the user of the system. ObjectIDs often include usernames, dates, or file names so that it is easier for users to remember what the LSID refers to. All objects that have LSIDs also have a Name property that commonly translates into the ObjectID portion of the LSID. The Name property of an object serves as the label for the object on most LabKey Server pages. It's a good idea to replace special characters such as spaces and punctuation characters with underscores or periods in the ObjectID.RevisionIDLabKey Server does not currently generate RevisionIDs in LSIDs, but can accept LSIDs that contain them.
This LSID identifies a specific protocol for a procedure called biotinylation. This LSID was created on a system with the LSID authority set to labkey.com. The namespace portion indicates that Protocol is the base type of the object, and the suffix value of Folder-2994 is added so that the same protocol can be loaded in multiple folders without a key conflict (see the discussion on substitution templates below). The ObjectId portion of the LSID can be named in whatever way the creator of the protocol chooses. In this example, the two-part ObjectId is based on a sample preparation stage (SamplePrep), of which one specific step is biotinylation (Biotinylation).
The extensive use of LSIDs in LabKey Server requires a system for generating unique LSIDs for new objects. LSIDs must be unique because they are used as keys to identify records in the database. These generated LSIDs should not inadvertently clash for two different users working in separate contexts such as different folders. On the other hand, if the generated LSIDs are too complex – if, for example, they guarantee uniqueness by incorporating large random numbers – then they become difficult to remember and difficult to share among users working on the same project.
LabKey Server allows authors of experiment description files (xar.xml files) to specify LSIDs which include substitution template values. Substitution templates are strings of the form
${<substitution_string>}
where <substitution_string> is one of the context-dependent values listed in the table below. When an experiment description file is loaded into the LabKey Server database, the substitution template values are resolved into final LSID values. The actual values are dependent on the context in which the load occurs.
Unless otherwise noted, LSID substitution templates are supported in a xar.xml file wherever LSIDs are used. This includes the following places in a xar.xml file:
The LSID value of the rdf.about attribute. You can use a substitution template for newly created objects or for references to objects that may or may not exist in the database.
References to LSIDs that already exist, such as the ChildProtocolLSID attribute.
Templates for generating LSIDs when using the ExperimentLog format (ApplicationLSID, OuputMaterialLSID, OutputDataLSID).
A limited subset of the substitution templates are also supported in generating object Name values when using the ExperimentLog format (ApplicationName, OutputMaterialName, and OutputDataName). These same templates are available for generating file names and file directories (OutputDataFile and OutputDataDir). Collectively these uses are listed as the Name/File ProtocolApplication templates in the table below.
Note: The following table lists the primitive, single component substitution templates first. The most powerful and useful substitution templates are compound substitutions of the simple templates. These templates are listed at the bottom of the table.
Table: LSID Substition Templates in LabKey Server
${LSIDAuthority}
Expands to
Server-wide value set on the Customize Site page under Site Administration. The default value is localhost.
Where valid
Any LSID
${LSIDNamespace.prefix}
Expands to
Base object name of object being identified by the LSID; e.g., Material, Data, Protocol, ProtocolApplication, Experiment, or ExperimentRun
Where valid
Any LSID
${Container.RowId} ${Container.path}
Expands to
Unique integer or path of project or folder into which the xar.xml is loaded. Path starts at the project and uses periods to separate folders in the hierarchy.
Where valid
Any LSID
Name/File ProtocolApplication templates
${XarFileId}
Expands to
Xar- + unique integer for xar.xml file being loaded
Where valid
Any LSID
Name/File ProtocolApplication templates
${UserEmail},${UserName}
Expands to
Identifiers for the logged-on user initiating the xar.xml load
Where valid
Any LSID
Name/File ProtocolApplication templates
${ExperimentLSID}
Expands to
rdf:about value of the Experiment node at the top of the xar.xml being loaded
The unque integer, LSID, and Name of the ExperimentRun being loaded
Where valid
LSID/Name/File ProtocolApplication templates that are part of that specific ExperimentRun
${InputName},${InputLSID}
Expands to
The name and lsid of the Material or Data object that is the input to a ProtocolApplication being generated using ExperimentLog format. Undefined if there is not exactly one Material or Data object that is input.
Where valid
LSID/Name/File ProtocolApplication templates that have exactly one input, e.g., MaxInputMaterialPerInstance + MaxInputDataPerInstance = 1
The individual parts of an InputLSID, as defined above. The namespacePrefix is defined as the namespace portion up to but not including the first period, if any. The namepsaceSuffix is the remaining portion of the namespace after the first period.
Where valid
LSID/Name/File ProtocolApplication templates that have exactly one input, i.e., MaxInputMaterialPerInstance + MaxInputDataPerInstance = 1
${InputInstance},${OutputInstance}
Expands to
The 0-based integer number of the ProtocolApplication instance within an ActionSequence. Useful for any ProtocolApplication template that includes a fractionation step. Note that InputInstance is > 0 whenever the same Protocol is applied multiple times in parallel. OutputInstance is only > 0 in a fractionation step in which multiple outputs are generated for a single input.
Where valid
LSID/Name/File ProtocolApplication templates that are part of that specific ExperimentRun
See Data object in next section for behavior and usage
Where valid
Any Data LSID only
Common Usage Patterns
In general, the primary object types in a Xar file use the following LSID patterns:
Experiment, ExperimentRun, Protocol
These three object types typically use folder-scoped LSIDs that look like
${FolderLSIDBase}:Name_without_spaces
In these LSIDs the object name and the LSID’s objectId are the same except for the omission of characters (like spaces) that would get encoded in the LSID.
ProtocolApplication
A ProtocolApplication is always part of one and only one ExperimentRun, and is loaded or deleted with the run. For ProtocolApplications, a run-scoped LSID is most appropriate, because it allows multiple runs using the same protocol to be loaded into a single folder. A run-scoped LSID uses a pattern like
${RunLSIDBase}:Name_without_spaces
Material
Material objects can be divided into two types: starting Materials and Materials that are created by a ProtocolApplication. If the Material is a starting material and is not the output of any ProtocolApplication, its scope is outside of any run. This type of Material would normally have a folder-scoped LSID using ${FolderLSIDBase}. On the other hand, if the Material is an output of a ProtocolApplication, it is scoped to the run and would get deleted with the run. In this case using a run-scoped LSID with ${RunLSIDBase} would be more appropriate.
Data
Like Material objects, Data objects can exist before any run is created, or they can be products of a run. Data objects are also commonly associated with physical files that are on the same file share as the xar.xml being loaded. For these data objects associated with real existing files, it is important that multiple references to the same file all use the same LSID. For this purpose, LabKey Server provides the ${AutoFileLSID} substitution template, which works somewhat differently from the other substitution templates. An ${AutoFileLSID} always has an associated file name on the same object in the xar.xml file:
If the ${AutoFileLSID} is on a starting Data object, that object also has a DataFileUrl element.
If the ${AutoFileLSID} is part of a XarTemplate.OutputDataLSID parameter, the XarTemplate.OutputDataFile and XarTemplate.OutputDataDir specify the file
If the ${AutoFileLSID} is part of a DataLSID (reference), the DataFileUrl attribute specifies the file.
When the xar.xml loader finds an ${AutoFileLSID}, it first calculates the full path to the specified file. It then looks in the database to see if there are any Data objects in the same folder that already point to that file. If an existing object is found, that object’s LSID is used in the xar.xml load. If no existing object is found, a new LSID is created.
You can use a data-integrated Electronic Lab Notebook (ELN) within LabKey LIMS, Biologics LIMS, and the Professional Edition of Sample Manager to efficiently document your research seamlessly integrated with your samples, assay data, files and other entity data. Refine how your work is presented and have it reviewed and approved in a user-friendly application that is easy to learn and use.Learn more in this section:
LabKey Server offers the functionality of a simple notebook or lab data management sysetm within a customizable folder, including:
Documentation of experimental procedures and results
Management of file-based content
Organization of lab notes and observations
Like any content on LabKey Server, lab resources are searchable via full-text search and securely shareable with individuals or groups of your choice.Each lab team can use a customized individual folder and you can separate functions on individual tabs within a folder. Within each folder or tab a combination of tools may be used for the lab notebook, including:
Workbooks: Use workbooks as containers for experimental results, work areas for uploading files, and holding lab notes.
Assays: Use assay designs to hold your instrument data in a structured format.
Sample Types: Use sample types to track and organize the materials used in laboratory experiments, including their derivation history.
Assay Request Tracker: A workflow designed especially for the assaying of specimens and samples, providing a collaboration tool that ties together the various elements of the work to be completed in a single trackable ticket.
Audit History: LabKey Server's extensive audit log can help you meet compliance challenges.
See the Lab Workflow Folder Tutorial to begin building a basic lab notebook. The tutorial shows you how to model a basic lab workflow which links together freezer inventories, samples, and downstream assay results. Use this example application as a starting point to extend and refine for your own workflows.
This tutorial shows you how to translate real entities in the lab into data objects in LabKey Server. The activities represented include:
Tracking sample vials and their freezer locations
Capture of experimental/assay result data
Linkage between vials and downstream experimental data
Basic lab workflows, such as the intake and processing of sample vials
Vial status, such as "Consumed", "Ready for processing", "Contaminated", etc.
A live, completed version of this tutorial is available here: Example Lab Workflow Folder.The following illustration shows a basic lab workflow, where samples, stored in vials inside of freezers, are run through an assay instrument, which outputs result data. The top portion of the image shows the real-world lab entities (freezers, vials, instruments, and Excel files); the bottom portion of the image shows the analogous LabKey Server data capture mechanisms (Lists, Sample Types, Assay Types, and Assay Designs).
This step explains how to put into place the tabs and web parts that form the user interface for the laboratory workflow folder.
Set Up a Folder
Log in to your server and navigate to your "Tutorials" project. Create it if necessary.
If you don't already have a server to work on where you can create projects, start here.
If you don't know how to create projects and folders, review this topic.
Create a new subfolder named "Lab Workflow". Choose the folder type "Assay."
Tabs
We will set up three tabs to reflect the basic workflow: the lab begins with Vials, which are run through different Experiments, which finally provide Assay Results.
Click Exit Admin Mode to hide the tab and web part editing tools.
Now you have a basic user interface: each tab represents a different part of the lab workflow in left to right order: vials --> experiments --> assay results.
In this step we import two different, but related, data tables to the Lab Workflow tutorial folder you just created:
Plasma sample inventory: a spreadsheet that describes different vials of plasma.
Assay result data: a spreadsheet that holds the experimental results of assaying the plasma.
These tables are connected by the fact that the assay data describes properties of the plasma in the vials. We will capture this relationship in the next step when we link these two tables together using a LabKey Server device called a lookup. In the current step, we will simply import these tables to the server.
Add Sample Type - Plasma
We will import information about the individual sample vials, such as the tissues stored and the barcode of each vial, like the table shown below.Sample Type - Plasma
Name
Sample Type
Flag
Status
LocationId
vl-100
Plasma
Received
i-123456
vl-200
Plasma
Flagged for review
Results Invalid
i-123457
vl-300
Plasma
Job Assigned
i-123458
vl-400
Plasma
In Transit
i-123459
vl-500
Plasma
Contaminated
i-123460
vl-600
Plasma
Preserved
i-123461
vl-700
Plasma
Results Verified
i-123462
Download the sample file Plasma.xlsx from this page.
Click the Vials tab.
On the Sample Types panel, click New Sample Type.
Provide the Name: "Plasma"
You do not need to provide a Naming Pattern since the data already contains a unique identifier in a "Name" column.
You don't need to add a description or parent column import alias.
Click the Fields panel to open it.
Click Manually Define Fields.
Click Add Field to add each of the following fields:
Status
LocationId
Click Save.
Now you will see Plasma included on the Sample Type web part.
Click Plasma to open it.
Click Import More Samples.
Click Upload file (.xslx, .xls, .csv, .txt) to open the file upload panel.
Leave the Import Options box for "update" unchecked.
Click Browse and select the Plasma.xlsx file you just downloaded.
Click Submit.
Import Assay Data - Immune Scores
Next, we import the data generated by assays performed on these plasma vials. The data includes:
Participant IDs - The subjects from which the samples were drawn.
Sample IDs - Note these values match the Name column in the Plasma table. This fact makes it possible to link these two tables (Assay and Sample Type) together.
Experimental measurements - Columns M1, M2, M3.
ParticipantId
SampleId
Date
M1
M2
M3
pt-1
vl-100
4/4/2020
1
0.8
0.12
pt-1
vl-200
6/4/2020
0.9
0.6
0.22
pt-1
vl-300
8/4/2020
0.8
0.7
0.32
pt-2
vl-400
4/4/2020
0.77
0.4
0.33
pt-2
vl-500
6/4/2020
0.99
0.5
0.44
pt-2
vl-600
8/4/2020
0.98
0.55
0.41
pt-3
vl-700
4/4/2020
0.94
0.3
0.32
pt-3
vl-800
6/4/2020
0.8
0.77
0.21
Follow the instructions below to import the assay data into the server.
Download the file immune-score.xlsx from this page. This is the data-bearing file. It holds the data results of your experiment, that is, the values measured by the instrument.
Click the Experiments tab.
Drag-and-drop this file into the Files panel.
In the Files panel select the file immune-score.xlsx and click Import Data.
In the Import Data pop-up dialog, select Create New Standard Assay Design and click Import.
On the Assay Import page:
In the Name field, enter "Immune Scores"
Set the Location to Current Folder (Lab Workflow).
Click Begin Import.
On the Data Import: Batch Properties page, do not change any values and click Next.
On the Data Import: Run Properties and Data File page, do not change any values and click Save and Finish.
Click the Assay Results tab.
You will now see the Immune Scores assay that you just defined.
Assay Terminology - Detailed information on how the server captures assay data.
Step 3: Create a Lookup from Assay Data to Samples
This step creates a link, a "lookup" in LabKey terminology, between the Immune Scores assay data and the Plasma sample type, providing easy navigation between the two. The lookup links the SpecimenId column in the assay results with the Name column in the Plasma sample type, as illustrated below:
To create the lookup, follow the instructions below:
Click the Assay Results tab.
Click the Immune Scores assay name to open the runs page.
Click Manage Assay Design and select Edit assay design.
Scroll down and click the Results Fields section to open it.
In the SpecimenID row, click the data type dropdown, currently reading: Text.
Select Lookup.
Confirm the Target Folder is the current folder.
For Target Schema, select samples.
For Target Table, select Plasma (String).
To save changes to the assay design, click Save (in the lower right).
The lookup creates a link between the SpecimenId field and the Plasma sample type, creating a link between assay result data and the particular vial that produced the results.To test these links:
Open the data grid by clicking immune-score.xlsx (the Assay ID is the filename by default).
Click a value in the SpecimenId field. Notice that the links take you to a detailed dashboard describing the original sample vial.
This topic shows a few ways to use the Lab Workspace folder we built during the tutorial, extending its functionality. In particular, it shows you how to incorporate freezer locations and sample status information, interlinking related information in a user friendly way.
When you have completed these steps, the Workspace will include the following tables and relationships:
Using the Lab Workspace
Here are some ways you can use the Lab Workspace folder:
Importing New Samples
When new samples arrive at the lab, register/import them on the Vials tab. You can import them in one of two ways:
as a new Sample Type by clicking New Sample Type.
or as new records in an existing Sample Type by clicking the target Sample Type and then clicking Import More Samples.
Each sample must have a unique name. The server enforces unique names for each sample on the server and will not allow you to import two samples with the same name. The easiest way is to provide unique values in the Name (or SampleID) field. There are other options for providing (or generating) sample IDs, described in the topic Sample Naming Patterns.
Import Assay Results
New assay results can be imported using the Experiments tab.
Drag-and-drop any new files into the Files web part.
Once they have been uploaded, select the new files, click Import Data, and select the target assay design.
Notice the Usages column in the files web part: Entries here link to the assay design(s) into which the data has been imported. Click to open the result data for that usage.
Discover Which Vial Generated Assay Results
When samples and assay results have been imported, they are automatically linked together (provided that you use the same id values in the Name and SpecimenId fields). To navigate from assay results to the original vial:
Click the Assay Results tab.
Then click the assay design (like "Immune Scores").
If multiple runs are listed, you can:
Click an Assay ID to see results for a specific run.
Select one or more rows using the checkboxes, then click Show Results to see a subset of results.
Click View Results (above the grid) to see all results.
The SpecimenID field contains links. Each link navigates to a details page describing the original vial.
Inventory Tasks
After extending the folder as described below, you can use the FreezerInventory grid to query the inventory and answer questions like 'How many samples are in freezer A?'
Extending the Lab Workbook Folder
Here some ways you can extend the functionality of the Lab Workbook folder:
Add freezer/inventory tracking of the vials.
Add status tracking for individual vials, such as "In Transit", "Received", "Used", "Ready for Processing", etc.
Add links from a vial to the results that were generated by it.
Track Freezer/Inventory Locations
This inventory list records the freezer/location of the sample vials. Import the inventory list as follows:
On the Available Lists page, click Create New List.
In the List Designer, enter the Name "FreezerInventory".
Click the Fields section to open it.
Drag and drop the "FreezerInventory.xlsx" file into the target area.
From the Key Field Name dropdown, select InventoryID.
Scroll down and notice that the checkbox under Import Data is checked, meaning the data will be imported as the list is created.
Scroll further down and click Save to both create the list and import the data.
You'll see the FreezerInventory added to the available lists. Click the name to see the data in a grid.
Link Plasma to the FreezerInventory
Next you will create a lookup from the Plasma samples to the FreezerInventory list, making it easy to find a vial's location in the lab's freezers.
Click the Vials tab.
Click the name of the Plasma sample type, then click Edit Type.
Click the Fields section to open it.
In the row for the LocationId field, change the Data Type to Lookup.
In the Lookup Definition Options set:
Target Folder: Current folder
Target Schema: lists
Target Table: FreezerInventory (String).
Scroll down and click Save.
The Plasma samples now link to matching records in the Inventory table.
Note: Any future Sample Types you add can make use of the Inventory table by including a lookup field (such as Barcode or FreezerLocation) as a lookup that points to the Inventory list.
Track Sample Status
Vials have different states throughout a lab workflow: first they are received by the lab, then they are stored somewhere, later they are processed to generate result data. The following list of states is used to track the different events in a vial's life cycle in the lab. The list of status states is not fixed, you can modify it as best fits your lab workflow.
Download the list: Status.xlsx, which is a list of possible status settings.
Go to > Manage Lists.
On the Available Lists page, click Create New List.
In the List Designer, enter the Name "Status".
Click the Fields section to open it.
Drag and drop the "Status.xlsx" file into the target area.
From the Key Field Name dropdown, select Status.
Scroll down and notice that the checkbox under Import Data is checked, meaning the data will be imported as the list is created.
Scroll further down and click Save to both create the list and import the data.
Link Plasma to the Status List
Next you will create a lookup from the Plasma samples to the Status list, making it easy to find the vial's state with respect to the basic lab workflow.
Click the Vials tab.
Click the Plasma sample type, and click Edit Type.
Click the Fields section to open it.
In the row for the Status field, change the Data Type to Lookup.
In the Lookup Definition Options set:
Target Folder: Current folder
Target Schema: lists
Target Table: Status (String).
Scroll down and click Save.
The Plasma samples now link to matching records in the Status table.
Note: Any future sample types you add can make use of the Status table, by adding a lookup field in the same way: by converting an existing field in the sample type to a lookup that points to the Status list.
Improved User Interface
To save yourself clicking through the UI each time you want to see the Plasma samples and the Assay results, you can add these tables directly to the Vials and Assay Results tabs respectively.To add the Plasma table directly to the Vials tab:
Select "Show the contents of a specific query and view."
Query: Plasma
Leave other fields at their defaults.
Click Submit.
Click the Vials tab to return to the main view. The new panel shows the same grid as you would see if you clicked to the details page for the Plasma sample type.
To add the Assay grid directly to the Assay Results tab:
Click the Assay Results tab.
Using the selector on the left, add the web part: Assay Results.
On the Customize Assay Results page:
Assay: select "General: Immune Scores" (it may be selected by default).
Show button in web part: leave checked.
Click Submit.
Now the results for the assay are shown directly on the main tab.
You can now click Exit Admin Mode to hide the tools for adding web parts.
Improve Link from Assay Results to Vial Details
Currently, the SpecimenID field in the "Immune Scores Results" data displays a link to the original vial that generated the data. By default, these links take you to a details page for the vial, for example:
But this is somewhat of a dead end in the application. The problem is that the vial details page does not contain any useful links.To correct this, we will override the target of the link: we will redirect it to a more useful view of vial details, a details view that includes links into the inventory and status information. In particular, we will link to a filtered view of the Plasma sample type, like this:
Under Name and Linking Options, notice the URL text box.
By entering URL patterns in this box, you can turn values in the column into links.
In the URL text box, enter the following URL pattern. This URL pattern filters the sample type table to one selected SpecimenID, referenced by the token ${SpecimenID}.
The reagent module may require significant customization and assistance, so it is not included in standard LabKey distributions. Please contact LabKey to inquire about support options.
Introduction
The Reagent database helps you organize your lab's reagents. It can help you track:
Reagent suppliers
Current reagent catalog and expiration dates
Current reagent locations (such as a particular freezer, box, etc.)
Reagent lot numbers and manufacturers
This topic explains how to install and use the Reagent database, which is based on LabKey Server's reagent module.
Installation
Install
Acquire the reagent module (see note above)
Stop LabKey server
Copy the reagent.module file into your /modules directory.
Restart the server.
Setup
Create a new folder or navigate to a folder where you'd like to install the reagent database.
Select > Folder > Management and click the Folder Type tab.
Check the box for reagent and click Update Folder.
On the main folder page, enter > Page Admin Mode.
Choose Query from the Select Web Part dropdown menu and click Add.
On the Customize Query page, enter the following values:
Web Part Title: <leave blank>
Schema: "reagent"
Query and View: Select the radio button Show the contents of a specific table and view and, from the dropdown menu, select "Reagents".
Allow user to choose query?: "Yes"
Allow user to choose view?: "Yes"
Button bar position: "Top"
Click Submit.
Populate the database with data:
Select > Go To Module > Reagent.
Click initialize within the setup text to open the "reagent-initialize.view"
Click the Initialize button.
Return to the main folder by clicking the Folder Name link near the top of the page. You'll see the populated Reagent database in the Query web part you created.
Use the Reagent Database
Navigation
The reagent database contains the following tables:
Antigens
Labels
Lots
Manufacturers
Reagents
Species (holds species reactivity data)
Titrations
Vials
Click Query to navigate between the tables.
Customize Tables
You can add custom fields (for example, "Expires" or "Price") to the Lots, Reagents, Titrations, and Vials tables. Learn more about the field editor in this topic: Field Editor.
Add New Reagents and Lots
To add new reagents and lots to the database, first you need to add information to these tables:
Antigens
Labels
Manufacturers
Species (optional)
Navigate to each table above, select (Insert Data) > Insert New Row for each, and enter the appropriate information in the insert forms.Next navigate to the Reagent table, select (Insert Data) > Insert New Row and use the Antigen, Label, and Species dropdowns to select the data you just entered above. (Note that you can select multiple species values.)Next navigate to the Lots table, select (Insert Data) > Insert New Row and use the Reagent and Manufacturer dropdowns to select the data.
Add New Vials and Titrations
To add new vials and titrations, first you need to add lot data, then navigate to the vials table and select (Insert Data) > Insert New Row.
Bulk Edits
To edit multiple rows simultaneously, place checkmarks next to the rows you wish to edit, and click Bulk Edit. Only changed values will be saved to the selected rows.
Samples
Samples are the raw materials (reagents, blood, tissue, etc.) or processed derivatives of these materials that are analyzed as part of experiments.A Sample Type (previously known as a Sample Set) is a group of samples accompanied by a suite of properties that describe shared characteristics of all samples in the group.
A sample type can be used to quickly apply shared properties to a group of samples instead of adding these properties to each sample individually.
Sample types can be scoped to a particular folder or project context, or shared site wide if they are defined in the Shared project.
Samples can be linked with downstream assay results, using a lookup field in the assay design to the sample type. For details, see Link Assay Data to Samples.
The derivation of a sample into aliquots, or samples that are mixtures of multiple parent samples, can be tracked. For details, see Sample Parents: Derivation and Lineage.
Samples vs. Specimens
The terms sample and specimen refer to two different methods of tracking the same types of physical materials (such as blood draws or tissue).
Specimens. LabKey's specimen infrastructure is provided as a legacy option for use with the study module, enabling close integration of specimen information with other types of study data. The module provides a specimen request and tracking system for specimens. However, specimen information imported to LabKey Server must conform to a constrained format with a defined set of fields.
Samples. Samples are less constrained than specimens. Administrators can define sample properties and fields tailored to their particular experiments. Sample infrastructure is provided by LabKey's experiment module. It supports tracking the derivation history of these materials but does not directly support request tracking. Samples are used by LabKey's Flow, MS2 and Microarray modules.
A Sample Type is a group of samples accompanied by a suite of properties that describe shared characteristics of all samples in the group. Users must have administrator permissions (the Folder Admin role, or higher) to create new Sample Types or edit existing Sample Type designs.Before you create a new Sample Type, consider how you will provide or generate unique Sample IDs. For options, see Sample Naming Patterns.Decide also where you want the Sample Type to be available. If you define it in a given folder, it will be available only in that folder. Defined in a project, you will be able to add samples to it from the project itself and all subfolders of that project. If you define it in the Shared project, it will be available site-wide.
In the Sample Types web part, click New Sample Type.
Sample Properties
Enter:
Name: (Required) This is the name for the Sample Type itself, not to be confused with a "Name" column within the Sample Type for unique names for the Samples.
The name must be unique, must start with a letter or number character, and cannot contain special characters or some reserved substrings listed here.
Description: Optional description of the Sample Type itself. This is not to be confused with the "Description" column also available in the sample data.
Naming Pattern: Every sample must have a unique name/id within the Sample Type. You can either provide these yourself in a "Name" column in your data, or ask the server to generate them using a naming pattern provided here. If you are providing names, leave the naming pattern field blank and ignore the placeholder text. Learn more about the options in the topic Sample Naming Patterns.
Add Parent Alias: (Optional) If your imported data will include lineage/parentage information for your samples, provide the column name and Sample Type here. Learn more in this topic: Sample Parents: Derivation and Lineage.
Auto-link Data to Study: (Optional) Select a study to which the data in this Sample Type should be linked. Learn more in this topic: Link Sample Data to Study.
Linked Dataset Category: (Optional) Specify the desired category for the Sample Dataset that will be created (or appended to) in a target study when rows are linked. Learn more here: Manage Categories.
Barcodes: (Optional) To enable auto-generation of barcodes for these samples, you will add a field of type Unique ID in the Fields section. Learn more in this topic: Barcode Fields.
Sample Fields
Default System Fields
Every Sample Type includes Default System Fields, including:
Name: Unique ID provided or generated from a naming pattern
SampleState: Represents the status of the sample
Description: Contains a description for this sample
MaterialExpDate (Expiration Date): The date this sample expires on. Learn more about using Expiration Dates in the Sample Manager Documentation
StoredAmount (Amount): The amount of this sample
Units: The units associated with the Amount value for this sample
AliquotCount (Aliquots Created Count)
Learn more about providing the Storage Amount and Units at the time of Sample creation in this topic: Provide Sample Amount and Units
Click the Fields panel to open it.
The Default System Fields are shown in a collapsible panel.
You can use the Enabled checkboxes to control whether to use or hide eligible system fields. "Name" and "SampleState" are always enabled.
Use the Required checkbox if you want to make one of these fields required. By default only "Name" is required.
Click the to collapse the default fields section.
Custom Fields
To add additional custom fields, you have several options:
Import or infer fields from file: Select or drag and drop a file into the panel.
Infer from an example spreadsheet: Formats supported are: .csv, .tsv, .txt, .xls, or .xlsx.
Each sample in a Sample Type must have a unique name/id. If your data does not already contain unique identifiers for each row, the system can generate them upon import using a Naming Pattern (previously referred to as a name expression) as part of the Sample Type definition.
Administrators can customize how sample IDs are generated using templates that build a unique name from syntax elements including:
String constants
Incrementing numbers
Dates and partial dates
Values from the imported data file, such as tissue types, lab names, subject ids, etc.
For example, this simple naming pattern uses a string constant 'S', a dash, and an incrementing number:
S-${genId}
...to generate sample IDs:
S-1 S-2 S-3 and so on...
This example builds a sample ID from values in the imported data file, and trusts that the combination of these column values is always unique in order to keep sample IDs unique:
Naming patterns can incorporate the following elements. When surrounded by ${ } syntax, they will be substituted during sample creation; most can also accommodate various formatting options detailed below.
Name Element
Description
Scope0000000000000000000000000
genId
An incrementing number starting from 1. This counter is specific to the individual Sample Type or Data Class. Not guaranteed to be continuous.
Current Sample Type / Data Class in a given container
sampleCount
A counter incrementing for all samples of all Sample Types, including aliquots. Not guaranteed to be continuous. Learn more below.
All Sample Types in the application (top level project plus any subfolders)
rootSampleCount
A counter incrementing for non-aliquot samples of all Sample Types. Not guaranteed to be continuous. Learn more below.
All Sample Types in the application (top level project plus any subfolders)
Loads data from some field in the data being imported. For example, if the data being imported has a column named "ParticipantID", use the element/token "${ParticipantID}"
Current Sample Type / Data Class
dailySampleCount
An incrementing counter, starting with the integer '1', that resets each day. Can be used standalone or as a modifier.
All Sample Types / Data Classes on the site
weeklySampleCount
An incrementing counter, starting with the integer '1', that resets each week. Can be used standalone or as a modifier.
All Sample Types / Data Classes on the site
monthlySampleCount
An incrementing counter, starting with the integer '1', that resets each month. Can be used standalone or as a modifier.
All Sample Types / Data Classes on the site
yearlySampleCount
An incrementing counter, starting with the integer '1', that resets each year. Can be used standalone or as a modifier.
All Sample Types / Data Classes on the site
randomId
A four digit random number for each sample row. Note that this random number is not guaranteed to be unique.
Current Sample Type / Data Class
batchRandomId
A four digit random number applied to the entire set of incoming sample records. On each import event, this random batch number will be regenerated.
Current Sample Type / Data Class
Inputs
A collection of all DataInputs and MaterialInputs for the current sample. You can concatenate using one or more values from the collection.
Current Sample Type / Data Class
DataInputs
A collection of all DataInputs for the current sample. You can concatenate using one or more values from the collection.
Current Sample Type / Data Class
MaterialInputs
A collection of all MaterialInputs for the current sample. You can concatenate using one or more values from the collection.
Current Sample Type / Data Class
Format Number Values
You can use formatting syntax to control how the tokens are added. For example, "${genId}" generates an incrementing counter 1, 2, 3. If you use a format like the following, the incrementing counter will have three digits: 001, 002, 003.
When you are using a data column in your string expression, you can specify a default to use if no value is provided. Use the defaultValue string modifier with the following syntax. The 'value' argument provided must be a String in ' single straight quotes. Double quotes or curly quotes will cause an error during sample creation.
${ColumnName:defaultValue('value')}
Use String Modifiers
Most naming pattern elements can be modified with string modifiers, including but not limited to:
:date - Use only the date portion of a datetime value.
:date('yy-MM-dd') - Use only the date portion of a datetime value and format it with the given format.
:suffix('-') - Apply the suffix shown in the argument if the value is not null.
:join('_') - Combine a collection of values with the given argument as separator.
:first - Use the first of a series of values.
The supported options are described in this topic:
${DataInputs:join('_'):defaultValue('S')} means 'Join together all of the DataInputs separated by underscores, but if that is null, then use the default: the letter S'
Incorporate Lineage Elements
To include properties of sample sources or parents in sample names, use lookups into the lineage of the sample.
Specific data type inputs: MaterialInputs/SampleType1/propertyA, DataInputs/DataClass2/propertyA, etc.
Import alias references: parentSampleA/propertyA, parentSourceA/propertyA, etc.
In some scenarios, you may be able to use a shortened syntax referring to an unambiguous parent property: Inputs/propertyA, MaterialInputs/propertyA, DataInputs/propertyA
This option is not recommended and can only be used when 'propertyA' only exists for a single type of parent. Using a property common to many parents, such as 'Name' will produce unexpected results.
For example, to include source metadata (e.g. my blood sample was derived from this mouse, I would like to put the mouse strain in my sample name), the derived sample's naming pattern might look like:
Blood-${DataInputs/Mouse/Strain}-${genId}
You can use the qualifier :first to select the first of a given set of inputs when there might be several.If there might be multiple parent samples, you could choose the first one in a naming pattern like this:
Blood-${parentBlood/SampleID:first}-${genId}
Include Ancestor Names/Properties Using "~"
To reference an ancestor name (or other property), regardless of the depth of the ancestry, you can use syntax that includes a tilde and will 'walk the lineage tree' to the named Source or Sample Type regardless of depth of a lineage tree (up to a maximum depth of 20). Note that this type of syntax may be more resource intensive, so if you know that the ancestor will always be the direct parent or at another specific/consistent level, you should use another lineage lookup for efficiency.For example, consider a "Participant" Source Type and also a Sample Type like "Blood" that could be either a direct 'child' of the source, or a grandchild (of an intermediate sample like "Tissue"), or any further descendent. You can include properties of the Participant source of a "Blood" sample with a naming pattern like this:
This syntax can be combined with other naming pattern elements, including counters as shown in this example. This will maintain a counter per Participant, regardless of the depth of tree where the sample is created:
${${~DataInputs/Participant/Name}-:withCounter}
Note that if the ancestor type appears multiple times in the lineage for a given sample, the "furthest" ancestor will be used.
Include Grandparent Names/Properties Using ".."
If you know that the ancestor of interest is a fixed number of generations above the direct parent, i.e. the grandparent or great-grandparent generation, you can use ".." syntax to walk the tree to that specific level. Here are a few examples of syntax for retrieving names from the ancestry of a sample. For simplicity, each of these examples is shown followed by a basic ${genId} counter, but you can incorporate this syntax with other elements. Note that the shortened syntax available for first generation "parent" lineage lookup is not supported here. You must specify both the sample type and the "/name" field to use.To use the name from a specific grandparent sample type, use two levels:
To define a naming pattern that uses the name of the grandparent of any type, you can omit the grandparent sample type name entirely. For example, if you had Plasma samples that might have any number of grandparent types, you could use the grandparent name using syntax like any of the following:
It is possible to include commas in Sample and Bioregistry (Data Class) entity names, though not a best practice to do so. Commas are used as sample name separators for lists of parent fields, import aliases, etc., so names containing commas have the potential to create ambiguities.If you do use commas in your names, whether user-provided or LabKey-generated via a naming pattern, consider the following:
To add or update lineage via a file import, you will need to surround the name in quotes (e.g, "WC-1,3").
To add two parents, one with a comma, you would only quote the comma-containing name, thus the string would be: "WC-1,3",WC-4.
If you have commas in names, you cannot use a CSV or TSV file to update values. CSV files interpret the commas as separators and TSV files strip the quotes 'protecting' commas in names as well. Use an Excel file (.xlsx or .xls) when updating data for sample names that may include commas.
Naming Pattern Examples
Naming Pattern
Example Output
Description
${genId}
1 2 3 4
a simple sequence
S-${genId}
S-1 S-2 S-3 S-1
S- + a simple sequence
${Lab:defaultValue('Unknown')}_${genId}
Hanson_1 Hanson_2 Krouse_3 Unknown_4
The originating Lab + a simple sequence. If the Lab value is null, then use the string 'Unknown'.
S_ + the current date + daily resetting incrementing integer
S_${Column1}_${Column2}
S_Blood_PT101
Create an id from the letter 'S' and two values from the current row of data, separated by underscore characters.
${OriginalCellLine/Name}-${genId}
CHO-1, CHO-2
Here OriginalCellLine is a lookup field to a table of orginating cell lines. Use a slash to 'walk the lookup'. Useful when you are looking up an integer-keyed table but want to use a different display field.
Incorporate Sample Counters
Sample counters can provide an intuitive way to generate unique sample names. Options include using a site-wide, container-wide, date-specific, or other field value-specific counter for your sample names.
See and Set genId
genId is an incrementing counter which starts from 1 by default, and is maintained internally for each Sample Type or Data Class in each container. Note that while this value will increment, it is not guaranteed to be continuous. For example, any creations by file import will 'bump' the value of genId by 100, as the system is "holding" a batch of values to use.When you include ${genId} in a naming pattern, you will see a blue banner indicating the current value of genId.If desired, click Edit genId to set it to a higher value than it currently is. This action will reset the counter and cannot be undone.As a special case, before any samples have been created, you'll see a Reset GenId as well as the Edit GenId button. This can be used to revert the change to GenID, but note that it is not available once any samples of this type have been created.
sampleCount Token
When you include ${sampleCount} as a token in your naming pattern, it will be incremented for every sample created in the application (a top-level project and any sub-projects it contains), including aliquots. This counter value is stored internally and continuously increments, regardless of whether it is used in naming patterns for the created samples.For example, consider a system of naming patterns where the Sample Type (Blood, DNA, etc) is followed by the sampleCount token for all samples, and the default aliquot naming pattern is used. For "Blood" this would be:
A series of new samples and aliquots using such a scheme might be named as follows:
Sample/Aliquot
Name
value of the sampleCount token
sample
Blood-1000
1000
aliquot
Blood-1000-1
1001
aliquot
Blood-1000-2
1002
sample
DNA-1003
1003
aliquot
DNA-1003-1
1004
sample
Blood-1005
1005
If desired, you could also use the sampleCount in the name of aliquots directly rather than incorporating the "AliquotedFrom" sample name in the aliquot name. For Blood, for example, the two naming patterns could be the same:
Blood-${sampleCount} <- for samples Blood-${sampleCount} <- for aliquots
In this case, the same series of new samples and aliquots using this naming pattern convention would be named as follows:
Sample/Aliquot
Name
sample
Blood-1000
aliquot
Blood-1001
aliquot
Blood-1002
sample
DNA-1003
aliquot
DNA-1004
sample
Blood-1005
The count(s) stored in the sampleCount and rootSampleCount tokens are not guaranteed to be continuous or represent the total count of samples, much less the count for a given Sample Type, for a number of reasons including:
Addition of samples (and/or aliquots) of any type anywhere in the application will increment the token(s).
Any failed sample import would increment the token(s).
Import using merge would increment the token(s) by more than by the new number of samples. Since we cannot tell if an incoming row is a new or existing sample for merge, the counter is incremented for all rows.
Administrators can see the current value of the sampleCount token on the Administration > Settings tab. A higher value can also be assigned to the token if desired. You could also use the :minValue modifier in a naming pattern to reset the count to a higher value.
rootSampleCount Token
When you include ${rootSampleCount} as a token in your naming pattern, it will be incremented for every non-aliquot (i.e. root) sample created in the application (a top-level project and any subprojects it contains). Creation of aliquots will not increment this counter, but creation of any Sample of any Sample Type will increment it.For example, if you use the convention of using the Sample Type (Blood, DNA, etc.) followed by the rootSampleCount token for all samples, and the default aliquot naming pattern, for "Blood" this would be:
Several sample counters are available that will be incremented based on the date when the sample is inserted. These counters are incrementing, but since they apply to all sample types and data classes across the site, within a given sample type or data class, values will be sequential but not necessarily contiguous.
dailySampleCount
weeklySampleCount
monthlySampleCount
yearlySampleCount
All of these counters can be used in either of the following ways:
As standalone elements of a naming pattern, i.e. ${dailySampleCount}, in which case they will provide a counter across all sample types and data classes based on the date of creation.
As modifiers of another date column using a colon, i.e. ${SampleDate:dailySampleCount}, in which case the counter applies to the value in the named column ("SampleDate") and not the date of creation.
Do not use both "styles" of date based counter in a single naming pattern. While doing so may pass the name validation step, such patterns will not successfully generate sample names.
:withCounter Modifier
Another alternative for adding a counter to a field is to use :withCounter, a nested substitution syntax allowing you to add a counter specific to another column value or combination of values. Using :withCounter will always guarantee unique values, meaning that if a name with the counter would match an existing sample (perhaps named in another way), that counter will be skipped until a unique name can be generated.The nested substitution syntax for using :withCounter is to attach it to an expression (such as a column name) that will be evaluated/substituted first, then surround the outer modified expression in ${ } brackets so that it too will be evaluated at creation time.This modifier is particularly useful when naming aliquots which incorporate the name of the parent sample, and the desire is to provide a counter for only the aliquots of that particular sample.
The default naming pattern for creating aliquots combines the value in the AliquotedFrom column (the originating Sample ID), a dash, and a counter specific to that Sample ID:
${${AliquotedFrom}-:withCounter}
You could also use this modifier with another column name as well as strings in the inner expression. For example, if a set of Blood samples includes a Lot letter in their name, and you want to add a counter by lot to name these samples, names like Blood-A-1, Blood-A-2, Blood-B-1, Blood-B-2, etc. would be generated with this expression. The string "Blood" is followed by the value in the Lot column. This combined expression is evaluated, and then a counter is added:
${Blood-${Lot}-:withCounter}
Use caution to apply the nested ${ } syntax correctly. The expression within the brackets that include the :withCounter modifier is all that it will be applied to. If you had a naming pattern like the following, it looks similar to the above, but would only 'count' the number of times the string "Blood" was in the naming pattern, ignoring the Lot letter, i.e. "A-Blood-1, A-Blood-2, B-Blood-3, B-Blood-4:
${Lot}-${Blood-:withCounter}
This modifier can be applied to a combination of column names. For example, if you wanted
a counter of the samples taken from a specific Lot on a specific Date (using only the date portion of a "Date" value, you could obtain names like 20230522-A-1, 20230522-A-2, 20230523-A-1, etc. with a pattern like:
${${Date:date}-${Lot}-:withCounter}
You can further provide a starting value and number format with this modifier. For example, to have a three digit counter starting at 42, (i.e. S-1-042, S-1-043, etc.) use:
${${AliquotedFrom}-:withCounter(42,'000')}
:minValue Modifier
Tokens including genId, sampleCount, and rootSampleCount can be reset to a new higher 'base' value by including the :minValue modifier. For example, to reset sampleCount to start counting at a base of 100, use a naming pattern with the following syntax:
S-${sampleCount:minValue(100)}
If you wanted to also format the count value, you could combine the minValue modifier with a number formatter like this, to make the count start from 100 and be four digits:
S-${sampleCount:minValue(100):number('0000')}
Note that once you've used this modifer to set a higher 'base' value for genId, sampleCount, or rootSampleCount, that value will be 'sticky' in that the internally stored counter will be set at that new base. If you later remove the minValue modifier from the naming pattern, the count will not 'revert' to any lower value. This behavior does not apply to using the :minValue modifier on other naming pattern tokens, where the other token will not retain or apply the previous higher value if it is removed from the naming pattern.
Advanced Options: XML Counter Columns
Another way to create a sequential counter based on values in other columns, such as an incrementing series for each lot of material on a plate, is to use XML metadata to define a new "counter" column paired with one or more existing columns to provide and store this incrementing value. You then use the new column in your naming pattern.For example, consider that you would like to generate the following sample type for a given plate:
Name (from expression)
Lot
SampleInLot
Plate Id
Blood-A-1
A
1
P1234
Blood-A-2
A
2
P1234
Blood-B-1
B
1
P1234
Blood-B-2
B
2
P1234
The "SampleInLot" column contains a value that increments from 1 independently for each value in the "Lot" column. In this example, the naming pattern would be S-${Lot}-${SampleInLot}. To define the "SampleInLot" column, you would create it with data type integer, then use XML metadata similar to this:
<tables xmlns="http://labkey.org/data/xml"> <table tableName="MySampType" tableDbType="NOT_IN_DB"> <javaCustomizer class="org.labkey.experiment.api.CountOfUniqueValueTableCustomizer"> <properties> <property name="counterName">SampleCounter</property> <property name="counterType">org.labkey.api.data.UniqueValueCounterDefinition</property> <!-- one or more pairedColumns used to derive the unique value --> <property name="pairedColumn">Lot</property> <!-- one or more attachedColumns where the incrementing counter value is placed --> <property name="attachedColumn">SampleInLot</property> </properties> </javaCustomizer> </table> </tables>
The following rules apply to these XML counters:
The counter you create is scoped to a single Sample Type.
The definition of the counter column uses one or more columns to determine the unique value. When you use the counter in a naming pattern, be sure to also include all the paired columns in addition to your counter column to ensure the sample name generated will be unique.
The incrementing counter is maintained within the LabKey database.
Gaps in the counter's sequence are possible because it is incremented outside the transaction. It will be sequential, but not necessarily contiguous.
When you insert into a sample type with a unique counter column defined, you can supply a value for this unique counter column, but the value must be equal to or less than the current counter value for the given paired column.
Naming Pattern Validation
During creation of a Sample Type, both sample and aliquot naming patterns will be validated. While developing your naming pattern, the admin can hover over the for a tooltip containing either an example name or an indication of a problem.When you click Finish Creating/Updating Sample Type, you will see a banner about any syntax errors and have the opportunity to correct them.Errors reported include:
Invalid substitution tokens (i.e. columns that do not exist or misspellings in syntax like ":withCounter").
Keywords like genId, dailySampleCount, now, etc. included without being enclosed in braces.
Mismatched or missing quotes, curly braces, and/or parentheses in patterns and formatting.
Use of curly quotes, when straight quotes are required. This can happen when patterns are pasted from some other applications.
Once a valid naming pattern is defined, users creating new samples or aliquots will be able to see an Example name in a tooltip both when viewing the sample type details page (as shown above) and when creating new samples in a grid within the Sample Manager and Biologics applications.
Caution: Escaping Special Characters
When field names or data type names contain special characters, including but not limited to {,}, and \, these characters must be escaped with a backslash \.
Caution: When Names Consist Only of Digits
Note that while you could create or use names/naming patterns that result in only digits, you may run into issues if those "number-names" overlap with row numbers of other entities. In such a situation, when there is ambiguity between name and row ID, the system will presume that the user intends to use the value as the name.
This topic is under construction for the 25.7 (July 2025) release. For the previous documentation of this feature, click here.
Aliquots can be generated from samples using LabKey Biologics and Sample Manager. As for all samples, each aliquot must have a unique sample ID (name). A custom Aliquot Naming Pattern can be included in the definition of a Sample Type. If none is provided, the default is the name of the sample it was aliquoted from, followed by a dash and counter.
To set a naming pattern, open your Sample Type design within LabKey Biologics or Sample Manager. This option cannot be seen or set from the LabKey Server interface.As with Sample Naming Patterns, your Aliquot Naming Pattern can incorporate strings, values from other columns, different separators, etc. provided that your aliquots always will have unique names. It is best practice to include the AliquotedFrom column, i.e. the name of the parent sample, but this is not strictly required by the system.
Aliquot Pattern Validation
During pattern creation, aliquot naming patterns will be validated, giving the admin an opportunity to catch any syntax errors. Users will be able to see the pattern and an example name during aliquot creation and when viewing sample type details. Learn more in this topic:
Using nested substitution syntax, you can include a counter specific to the value in the AliquotedFrom column (the originating Sample ID). These counters will guarantee unique values, i.e. will skip a count if a sample already exists using that name, giving you a reliable way to create unique and clear aliquot names.Prefix the :withCounter portion with the expression that should be evaluated first, then surround the entire expression with ${ } brackets. Like sample naming patterns, the base pattern may incorporate strings, and other tokens to generate the desired final aliquot name.The following are some examples of typical aliquot naming patterns using withCounter. Learn more in this topic: Sample Naming Patterns
Dash Count / Default Pattern
By default, the name of the aliquot will use the name of its parent sample followed by a dash and a counter for that parent’s aliquots.
${${AliquotedFrom}-:withCounter}
For example, if the original sample is S1, aliquots of that sample will be named S1-1, S1-2, etc. For sample S2 in the same container, aliquots will be named S2-1, S2-2, etc.
Dot Count
To generate aliquot names with "dot count", such as S1.0, S1.1, an admin can set the Sample Type's aliquot naming pattern to:
${${AliquotedFrom}.:withCounter}
Start Counter at Value
To have the aliquot counter start at a specific number other than the default 0, such as S1.1001, S1.1002., set the aliquot naming pattern to:
${${AliquotedFrom}.:withCounter(1001)}
Set Number of Digits
To use a fixed number of digits, use number pattern formatting. For example, to generate S1-001, S1-002, use:
${${AliquotedFrom}-:withCounter(1, '000')}
Include Lineage Elements
As for Sample IDs, Aliquot Naming Patterns can include elements like names and properties from the AliquotedFrom sample's lineage. Learn more about this syntax here:
When viewing the grid of a sample type, select (Insert Data) > Insert New Row from the grid menu.
Enter the properties of the new sample.
You can provide values for your custom properties, as well as the default system properties, including the expiration date and amount of the sample (with units).
Click Submit.
Import Bulk Sample Data
In the Sample Types web part, click the name of the sample type.
Click Import More Samples or Click (Insert Data) > Import Bulk Data to reach the same input page.
On the Import Data page:
Download Template: This button downloads a file template showing necessary and optional columns (fields) for your sample type.
Under Import Options during either file or copy/paste import, the default is Add samples. If any rows match existing samples, the import will fail.To Update samples, select this option and import data for existing samples. When you check the Allow new samples during update box, you can import and merge both existing and new sample rows.Use the update option to update lineage of samples, including parent information, inputs, sources, etc.
Change parent information: By including revised parent information, you can change any existing parent sample relationships.
Delete parents/sources: Deleting the parents of a sample is accomplished by updating existing sample data with parent columns empty.
Note that if a sample has more than one type of parent (either data classes or other samples), changing or deleting parents of one type will not change the relationships to other types of parents not specified in the merged data.
Import Using the API
Learn how to import Sample Type and Data Class data via the JavaScript API in this topic: Use the Registry API
Premium Resource AvailableSubscribers to Premium Editions of LabKey Server can find an example of splitting a large upload into chunks in this topic:
For any sample creation method, you can specify the storage amount and units, provided these default system fields are enabled. The StoredAmount column is labeled "Amount"; the Units field is labeled "Units". Importing data via a file will map a column named either "Amount" or "StoredAmount" to the StoredAmount field.Both the StoredAmount and Units fields have display columns attached to them and, when a Sample Type has a display unit defined (as is typical in both the Sample Manager and Biologics LIMS), the value displayed in the StoredAmount column will be the amount converted to that display unit. The Units column will show the display value if a sample type has one defined.Because this display column could prevent users from seeing the data as entered, we also provide two new columns "RawAmount" and "RawUnits", which present the data as stored in the database. These columns are hidden by default but can be added via customizing a samples grid.
Once you have a folder containing one or more sample types, you can view and manage them at several levels. This topic describes what you will find at each level.
You can add the Sample Types web part to any folder that has the Experiment module enabled. This web part lists all of the sample types available in the current folder.
Note the "Sample Count" field, which is a reserved, calculated column that shows the number of samples currently in a given sample type.
You can also hover over any row to reveal (edit) and (details) icons.
Delete Samples
To delete one or more samples, check the box for each of the desired row(s) and click the (Delete) icon. You will be asked to confirm the deletion and reminded that deletion cannot be undone.
Deletion Protection
It's important to note that samples cannot be deleted if they have derived sample or assay data dependencies, if they have a status that prevents deletion, or if they are referenced in workflow jobs or electronic lab notebooks.
If all rows selected for deletion have dependencies, the deletion will be denied.
If some rows have dependencies, a popup will explain that only the subset without dependencies can be deleted and you will have to confirm if you still want to delete them.
View an Individual Sample
Clicking on the name of any sample in the sample type brings you to a detailed view of the sample.The Sample Type Contents page shows you:
Standard properties. These are properties defined for a group of samples at the sample-type level.
Custom properties. These are properties of this individual sample.
Lineage. This section details the Parent Data, Precursor Samples, Child Data, and Child Samples.
Runs using this material or derived material. All listed runs use this sample as an input.
Edit a sample either by clicking the in the sample grid, or by clicking Edit on the details page. You can update the properties including the sample Name, however keep in mind that sample names must remain unique.
To create a link between your assay data and the sample that it describes, create a lookup column in the assay design that points to the appropriate Sample Type. Individual assay results can then be mapped to individual samples of that type.
When creating a lookup column that points to a sample type, you can choose to lookup either the RowID (an integer) or the Name field (a string). In most cases, you should lookup to the Name (String) field, since your assay data probably refers to the string id, not the integer RowID (which is a system generated value). Both fields are guaranteed to be unique within a sample type. When creating a new lookup column, you will see the same table listed twice with different types offered, Integer and String, as shown below.Choosing the Integer option will create a lookup on the RowId, choosing String will give you a lookup on the Name.
Importing Data Using Lookups
When importing assay data that includes a lookup field, the user will see a dropdown menu for that field allowing them to select the appropriate sample.
Note: if the sample type contains 10,000 rows or more, the lookup will not be rendered as a dropdown, but as a text entry panel instead.The system will attempt to resolve the text input value against the lookup target's primary key column or display value. For matching values, this will result in the same import behavior as if it were selected via dropdown menu.
Resolve Samples in Multiple Locations
[ Video Overview: Resolving Samples in Other Locations ]The lookup definition includes the target location. If you select a specific folder, the lookup only matches samples within that folder. If you leave the lookup target as the default location, the search for a match proceeds as follows:
First the current folder is searched for a matching name. If no match:
Look in the parent of the current folder, and if no match there, continue to search up the folder tree to the project level, or whereever the sample type itself is defined. If there is still no match:
Look in the Shared project (but not in any of its subfolders).
Note that while samples from parent/higher folders can be matched to assay data, samples in sibling or child folders will not be resolved.When you set the schema and table for the lookup, you can either target a specific sample type's query as shown above, or indicate that the server should look across all sample types by targeting the exp.material query.If multiple matches are found at any step in the search for a match to resolve the lookup, it will be reported to the user as an error.
Sample data can be integrated into a LabKey Study, aligning by participant and point in time. This is accomplished by linking the Sample data to the study, similar to how assay data can be linked to a study. Once connected, users can search for samples based on study subject details.
In order to align with study data, your sample data needs the concepts of participant and time. These are provided in fields of the following types. You can either provide values at the time of linking to study, or add fields of these types to your Sample Type:
Ordinary "DateTime" fields will not work for linking samples to studies.
You can name the fields as needed (i.e. "SubjectID" or "MouseID" for the Subject/Participant field) though specific naming rules apply if you plan to auto link samples to study.
Use Alignment Fields from a Parent Sample
To link samples where the participant and visit information is stored on a parent sample (i.e. to link a set of derivatives from a single blood draw from a participant), follow these steps:
The child sample type should not include fields of type Subject/Participant and VisitID/VisitDate.
The parent sample type does include the fields of type Subject/Participant and VisitID/VisitDate.
Lineage relationships are singular, meaning each child has only one parent of the parent type.
Edit the default grid view for the child sample type to include the participant and visit fields from the parent.
In the LabKey Server interface, open > Customize Grid for the child sample type.
Check Show Hidden Fields.
Expand the Input node, then Materials, then the node for the parent sample type.
Check the boxes for the participant and visit fields.
Save as the default view, being sure to check the Shared box to make this grid view available to all users.
Now return to the grid for the child sample type, noticing the participant and visit alignment fields.
Proceed to link the samples as if the fields had been defined on the child samples directly.
Link Samples to Study
View your grid of Samples. Select the rows to link, then click Link to Study.
Choose target study and click Next.
You can also specify a category for the linked dataset if desired. By default the new dataset will be in the "Uncategorized" category and can be manually reassigned later.
On the Verify Results page, provide ParticipantID and VisitID (or VisitLabel or VisitDate) if they are not already provided in your data:Click Link to Study.
Access Related Study Data
When the linking is complete, you will see the linked dataset in the study you specified. It looks very similar to the sample grid, but is a dataset in the study.Click a Participant ID to see more about that participant.Unless you specified a category during linking this new linked dataset will appear as "Uncategorized" on the Clinical and Assay Data tab and will be named for the Sample Type. Click the name to reopen the grid.Using grid view customization, you can add fields from other datasets in the study to get a full picture of the context for your sample data. For example, here we see sample barcodes alongside demographic information and CD4 levels recorded for the participant at that visit where that sample was drawn.In the linked dataset, you can also select one or more rows and click Recall to undo the linkage from here. This will not delete the source data from the sample type; it will undo the linkage so that row no longer appears in the study. Learn more in this topic: Recall Linked Data
Automatic Link-to-Study Upon Import
When you always plan to link samples to the same study, you can automate the manual process by building it directly into the Sample Type. When new samples are created, they will then automatically link to the target study using Participant and VisitID/VisitDate/VisitLabel information described. Similar to configuring this in assays, you can set this field during Sample Type creation, or edit later.Be sure to name the integration fields in your Sample Type to match your study target:
ParticipantID (type "Subject/Participant") or however this field is named in your study.
VisitID (type "Visit ID" or "Visit Label"): For linking to visit-based studies.
Date (type "Visit Date"): For linking to time-based studies. Note that a "DateTime" field will not work.
Shared Sample Types: Auto Link with Flexible Study Target
If your Sample Type design is defined at the project level or in the Shared project, it could potentially be used in many folders on your site.
You can set a single auto-link study destination as described above if all users of this sample type will consolidate linked data in a single study folder.
You may wish to configure it to always link data to a study, but not necessarily always to the same destination study. For this situation, choose the folder option "(Data import folder)". This will configure this Sample Type to auto link imported data to the study in the same folder where the data is imported.
Samples of this type imported throughout the scope where it is defined will now attempt to link to a study in the same folder.
In a study folder, imported sample data will automatically be linked to that study.
In a folder that is not a study, the sample import will still succeed, but an error message about the inability to link to the study that doesn't exist will be logged.
View Link to Study History
From the study dataset, click View Source Sample Type to return to the original Sample Type you linked. Click Link to Study History to see the history.Learn more about managing linking history in this topic: Link-To-Study History
Troubleshooting
Name Conflicts
In a study, you cannot have a dataset with the same name as an existing dataset or other table. This includes built in tables like "Cohort" that are unlikely to conflict, but in a scenario where you have a Sample Type that happens to have the same name as the "Subject Noun Singular" in the study, linking to study will fail when it attempts to create a new dataset with this conflicting name.For example, if you have a Sample Type named "Pig" and attempt to link to a study where the term used for the study subjects is also "Pig", you will see an error similar to:
Cannot invoke "org.labkey.api.data.TableInfo.getDefaultVisibleColumns()" because "tableInfo" is null
To work around this issue, you can rename the Sample Type to be something that does not conflict, such as "PigSamples".
LabKey Server understands relationships between different samples, including when a sample is an aliquot from a larger parent sample, or when a mixture is created from multiple parent samples. Samples can also be derived from Data Class sources as in the Biologics and Sample Manager applications. Visualizations of these relationships are shown using "Lineage Graphs".This topic covers how to capture and use these sample parent relationships in LabKey Server.
Derive Aliquots or Compounds Using the User Interface
You can derive samples using the user interface, either by splitting samples up into smaller portions to create aliquots or mixing multiple samples together to create compounds. The user interface for both operations is the same, the difference being that for aliquots you would derive from a single parent, and compounds would be composed of multiple samples.
From a Sample Type, select one (or more) sample(s) to use as parent, then click Derive Samples.
Specify Source Materials:
Name. The source samples are listed here.
Role. Roles allow you to label each input with a unique purpose.
Create new roles by selecting Add a New Role... and typing the role name in the box.
Roles already defined will be on the dropdown list alongside "Add a new role..."
Number of Derived Samples. You can create multiple derived samples at once.
Target Sample Type. You have the option to make derived samples part of an existing sample type. If you select no existing sample type, the derived output sample becomes part of the generic Materials table. You can access Materials from the Sample Types web part by clicking Show All Materials.
Click Next.
On the next page, you will see input sample information.
Enter a Name (required if no naming pattern is defined) and any properties specific to the Output Sample(s).
Click Submit.
The Roles specified become labels within the graphical view of the samples. To see the graph, click the link next to Lineage Graph:
There is also a link to Derive samples from this sample link on the details view of each individual sample, making creation of aliquots straightforward from here.
Capture Sample Parentage: Alias Parent Column
Within a Sample Type design, you can indicate one or more columns that contain parentage information. The parentage information can exist in the same or a different Sample Type. Parentage information can also exist in a Data Class. On import of new sample data, the Sample Type will look for the indicated column for the parentage information. Note that the column indicated is not actually added to the Sample Type as a new field, instead the system merely pulls data from indicated column to determine parentage relationships.For example, if you want to import the following samples table:
Name
ParentVial
v1
v1.1
v1
v1.2
v1
You would indicate the parent column alias as "ParentVial" (see screenshot below).To add parentage columns to a Sample Type:
Navigate to the folder or project where the Sample Type lives.
In the Sample Types web part, click the target Sample Type.
In the Sample Type Properties panel, click Edit Type.
On the Update Sample Type page, click Add Parent Alias.
In the Parent Alias enter the column name which contains the parentage information. The column name is case sensitive. The column name you enter here should not refer to a column in the existing Sample Type, instead it should refer to a column in the source data to be imported.
Use the drop down to indicate the Sample Type (or Data Class) where the parent samples reside. This may be in the same Sample Type "(Current Sample Type)", a different Sample Type, or a Data Class.
You can add additional parent aliases as needed.
Click Save.
When importing data, include the indicated parent column and its data, as in the example screenshot below.The system will parse the parentage information, and create lineage relationships. Learn about lineage graphs below.
Update Parent Information During Import
You can also update parent information during import (change or delete parents) by selecting the "Update data for existing samples during import" checkbox.Learn more in this topic: Update or Remove Sample Lineage During Import
Capture Sample Parentage: MaterialInputs and DataInputs
When importing samples, you can indicate parentage by including a column named "MaterialInputs/<NameOfSampleType>", where <NameOfSampleType> refers to some existing sample type, either a different sample type, or the current one. Similarly, "DataInputs/<NameOfDataClass>" can be used to refer to a parent from a data class.Note that the MaterialInputs (or DataInputs) column is not actually added to the Sample Type as a new field, instead the system merely pulls data from the column to determine parentage relationships. For example, the following indicates that DerivedSample-1 has a parent named M-100 in the sample type RawMaterials.
Name
MaterialInputs/RawMaterials
DerivedSample-1
M-100
You can point to parents in the same or different sample types. The following shows child and parent samples both residing in the sample type MySampleType:
Name
MaterialInputs/MySampleType
ParentSample-1
ChildSample-1
ParentSample-1
To indicate multiple parents, provide a list separated by commas. The following indicates that DerivedSample-2 is a mixture of two materials M-100 and M-200 in the RawMaterials sample type.
Name
MaterialInputs/RawMaterials
DerivedSample-2
M-100, M-200
You can indicate parents across multiple sample types by adding multiple MaterialInput columns. The following indicates that DerivedSample-3 has three parents, two from RawMaterials, and one from Reagents:
Name
MaterialInputs/RawMaterials
MaterialInputs/Reagents
DerivedSample-3
M-100, M-200
R-100
Samples can be linked to a parent from a data class member using a similar syntax. The following indicates that DerivedSample-4 is derived from an expression system ES-100:
Name
DataInputs/ExpressionSystems
DerivedSample-4
ES-100
Capture Sample Parentage: API
You can create a LABKEY.Exp.Run with parent samples as inputs and child derivatives as outputs.
Include Ancestor Information in Sample Grids
You can include fields from ancestor samples and sources by customizing grid views.For example, if you had a "PBMC" sample type and wanted to include in the grid both the "Draw Date" field from the parent "Blood" sample type AND the "Strain" field from a "Mouse" source, you would follow these steps:
Open the details page for the "PBMC" sample type.
Select > Customize Grid.
Expand the "Ancestors" node, then "Samples", then "Blood".
Check the box for "Draw Date".
In the "Ancestors" node, expand the "Registry And Sources" node, then "Mice".
Check the box for "Strain".
In the Selected Fields panel, drag the new fields to where you want them to be displayed, in this case to the right of the "Processing Operator" field.
You can use the to edit the title if desired, shown here we include the name of the parent type in the display title for each column.
Save the grid view, making sure to share it with all users.
You can decide whether to save it as the Default grid, or save it with a name that will let you find it under the menu.
Now the grid will include your fields. You could then filter, sort, and search based on that parent metadata. Shown below, we're showing only samples that come from "BALB/c" mice.
Include Aliquot Information in Sample Grids
When viewing a grid of samples, you will see the following calculated columns related to aliquots. Both of these columns are populated only for the original samples; they are set to null for aliquots themselves.
Aliquot Total Amount: Aggregates the total available amount of all aliquots and subaliquots of this sample. Includes any aliquot with a status of type "Available".
Aliquots Created Count: The total number of aliquots and subaliquots of this sample.
This column is populated only for samples, and set to zero for samples with no aliquots.
Like other columns, you can use a custom grid view to show, hide, or relocate them in the grid as desired.
Lineage SQL Queries
Syntax for querying the lineage of samples and dataclasses provides the opportunity to write SQL queries leveraging these relationships. The "expObject()" method available on some datastructures (Sample Types, Data Classes (i.e. Sources), exp.Runs, exp.Materials, exp.Data) returns a lineage object that can be used with the following variations of the IN/NOT IN operators. Using these operators on anything other than what expObject() returns will result in an error.
Operator
Action
IN EXPANCESTORSOF NOT IN EXPANCESTORSOF
Search the lineage object for all parents, grandparents, and older ancestors of the sample.
IN EXPDESCENDANTSOF NOT IN EXPDESCENDANTSOF
Search the lineage object for all children, grandchildren, and further descendants.
Example 1: Find Blood ancestors of a given PBMC sample:
SELECT Name FROM samples.Blood WHERE Blood.expObject() IN EXPANCESTORSOF (SELECT PBMC.expObject() FROM samples.PBMC WHERE PBMC.Name='PBMC-2390')
Example 2: Find PBMC descendants of a given Blood sample:
SELECT Name FROM samples.PBMC WHERE PBMC.expObject() IN EXPDESCENDANTSOF (SELECT Blood.expObject() FROM samples.Blood WHERE Blood.Name='Blood-1464')
Example 3: Find "Cell Line" (Registry Source) ancestors of all the samples linked from an assay run:
SELECT Name FROM exp.data.CellLine WHERE CellLine.expObject() IN EXPANCESTORSOF (SELECT Runs.expObject() FROM assay.General.MiniAssay.Runs WHERE Name = 'Run1')
Example 4: Find "Cell Line" (Registry Source) children of two Vectors:
PARAMETERS ( Vector1 VARCHAR DEFAULT 'Vector-1', Vector2 VARCHAR DEFAULT 'Vector-2' ) SELECT Q1.Name FROM (SELECT Name FROM samples.CellLines WHERE CellLines.expObject() IN EXPDESCENDANTSOF (SELECT Vectors.expObject() FROM exp.data.Vectors WHERE Vectors.Name=Vector1)) AS Q1 INNER JOIN (SELECT Name FROM samples.CellLines WHERE CellLines.expObject() IN EXPDESCENDANTSOF (SELECT Vectors.expObject() FROM exp.data.Vectors WHERE Vectors.Name=Vector2)) AS Q2 ON Q1.Name = Q2.Name
Lineage Graphs
Derived samples are represented graphically using "lineage graphs". To view:
Go to the Sample Type of interest.
Click the individual sample id or name.
In the Standard Properties panel, click the link for the Lineage Graph, i.e.: Lineage for <Your Sample Name>. If there is no parentage information, there will be no link and no graph.
Samples are represented as rectangles and the derivation steps are shown as diamonds. Note that elements in the graph are clickable links that navigate to details pages. By default you will see the Graph Detail View:If you click the Graph Summary View tab, you will see a different presentation of the same process and lineage.Clicking Toggle Beta Graph (New!) from the Graph Summary View will give you a another way to view the same information. Click any node to reset the point of focus of this graph, explore details in the panel to the right, and zoom/resize the graph using the controls.
Rendering Differences
Note that lineage graphs can differ depending on the way that the data is entered. When you manually derive multiple child samples from a parent via the Derive Samples button, the lineage graph summary view will show these child samples on one graph, as shown below in a case where two additional samples have been derived from the "Derived Yeast Sample".When the sample parent/child relationships are imported via copy-and-paste or via the API, separate lineage graphs will be rendered for each parent/child relationship. A single graph showing all the child sample simultaneously will not be available.
Premium Feature AvailableSubscribers to premium editions of LabKey Server can use the Run Builder to streamline creation of experiments with inputs and outputs:
Sample types help you track sample properties and laboratory workflows. The examples described in this topic are provide to help illustrate the features available. For a live version of these examples see the Sample Types Interactive Example.
A sample type recording the original blood draws and aliquots (derived child samples) for a clinical study. The original parent vials for the aliquots are indicated by the column "Parent Blood Sample" (previously known as "MaterialInputs").
Name
Description
Volume
VolumeUnit
SampleType
Parent Blood Sample
S1
Baseline blood draw
10
mL
Whole Blood
S2
Baseline blood draw
10
mL
Whole Blood
S3
Baseline blood draw
10
mL
Whole Blood
S4
Baseline blood draw
10
mL
Whole Blood
S1.100
Aliquot from original vial S1
2
mL
Whole Blood
S1
S1.200
Aliquot from original vial S1
2
mL
Whole Blood
S1
S1.300
Aliquot from original vial S1
6
mL
Whole Blood
S1
S2.100
Aliquot from original vial S2
2
mL
Whole Blood
S2
S2.200
Aliquot from original vial S2
4
mL
Whole Blood
S2
S2.300
Aliquot from original vial S2
4
mL
Whole Blood
S2
The following image shows the derivation graph for one the aliquot vials.
A sample type capturing cocktail recipes. The column "MaterialInputs/Cocktails" refers to multiple parent ingredient for each recipe.
Name
Description
MaterialType
MaterialInputs/Cocktails
Vodka
Liquor
Liquid
Gin
Liquor
Liquid
Bourbon
Liquor
Liquid
Bitters
Mixer
Liquid
Vermouth
Mixer
Liquid
Ice
Garnish
Solid
Olive
Garnish
Solid
Orange Slice
Mixer
Garnish
Martini
Classic Cocktail
Liquid
Gin, Vermouth, Ice, Olive
Vespers
Classic Cocktail
Liquid
Gin, Vodka, Vermouth, Ice
Old Fashioned
Classic Cocktail
Liquid
Bourbon, Bitters, Orange Slice
The derivation diagram for a Martini:
Beer Recipes
This example is consists of four different tables:
Beer Ingredient Types: a DataClass used to capture the different kinds of ingredients that go into beer, such as Yeast and Hops. In order to keep this example simple, we have consolidated all of the ingredient types into one large DataClass. But you could also split out each of these types into separate DataClasses, resulting in four different DataClases: Yeast Types, Hops Types, Water Types, and Grain Types.
Beer Recipe Types: a DataClass used to capture the different kinds of beer recipes, such as Lager, IPA, and Ale.
Beer Ingredient Samples: a Sample Type that instantiates the ingredient types.
Beer Samples: a Sample Type that captures the final result: samples of beer mixed from the ingredient samples and recipes.
The image below shows the derivation diagram for an Ale sample:Tables used in the sample:
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
This topic covers options for using barcode values with your samples. Existing barcode values in your data can be represented in a text or integer field, or you can have LabKey generate unique barcode identifiers for you by using a UniqueID field. LabKey-generated barcodes are read-only and unique across the server.Once a sample (or other entity) has either type of Barcode Field, you'll be able to search for it using these values.
To support barcodes generated and managed by LabKey, you will need a field of type "UniqueID" in your Sample Type. When you create a new Sample Type, you will be prompted to include this field.To add a UniqueID field to an existing Sample Type, open it, then click Edit Type. In the Sample Type Properties section, you will see under Barcodes, a blue banner inviting you to create the field necessary. Click Yes, Add Unique ID Field.By default, it will be named "Barcode". If you wish, you can click the Fields section to open it and edit the field name.Click Save. Barcodes will be automatically generated for any existing samples of this type, as described in the next section.
Generate UniqueID Barcodes
When you add a UniqueID field to a Sample Type that already contains samples, as you finish updating the sample type, you will be prompted to confirm that adding this field will generate barcodes for existing samples.Click Finish Updating Sample Type to confirm.In addition, when any new samples are added to this Sample Type by any method, barcodes will be generated for them. You cannot provide values for a UniqueID field and will not see them in a single row insert panel or in the template for importing a spreadsheet. When entering multiple rows in a grid, you will see a grayed out column indicating that barcodes will be generated and cannot be provided.UniqueID generated barcodes are 9+ digit text strings with leading zeros ending in an incrementing integer value. Ex: 000000001, 000000002, etc. Generated barcodes are unique across the server, i.e. if you use UniqueID barcodes for several different Sample Types or in different containers, every sample in the system will have a unique barcode. When more than a billion samples are defined, the barcode will continue to increment to 10 digits without leading zeros.Once generated by the system, barcodes in a UniqueID field cannot be edited and if data is imported into one of these fields, it will be ignored and an automatic barcode will be generated. If you need to provide your own barcode values or have the ability to edit them, do not use a UniqueID field.
Use Existing Barcode Values
If you have barcodes in your sample data already, and do not want them generated or managed by LabKey, you can include a field for them in your Sample Type. Choose field type "Text" or "Integer", and check the Barcode Field box for "Search this field when scanning samples".This field may be named "Barcode" if you like, but will not be managed by LabKey. It will have the "scannable" field property set to true.Users can locate samples by the values in this column, but must manage uniqueness and generate new barcode values for new samples outside of the application and provide them when creating (or editing) sample data.
Search by Barcode
Once your samples have a populated barcode column of either type, you can search for the value(s) to locate the sample(s). You can search, sort, and filter the samples grid, or use the global search bar in the header throughout LabKey Server.To more easily find samples by barcode value in the LabKey Biologics and Sample Manager applications, you can use the Find Samples by Barcode option.Enter values by typing or using a scanner, then click Find Samples to search for them.Learn more in the Sample Manager documentation:
Data Classes are used to represent virtual entity definitions that can be customized and defined by administrators. Members of a Data Class can be connected to physical samples using LabKey's existing experiment framework, or using Sample Manager and LIMS products, they are represented as "Sources", supporting lineage relationships with samples.For example, an administrator might define Data Classes to represent entities such as Protein Sequences, Nucleotide Sequences, Cells Lines, and Expression Systems. A Sample Type may be derived from an Expression System Data Class, which in turn is derived from Cell Line and Vector Data Classes. See LabKey Data Structures.
Data Classes support the concept of parentage/lineage. When importing data into a Sample Type, to indicate a Data Class parent, provide a column named "DataInputs/<NameOfDataClass>", where <NameOfDataClass> is some Data Class. Values entered under this column indicate the parent the sample is derived from. You can enter multiple parent values separated by commas. For example to indicate that sample-1 has three parents, two in DataClassA, and one in DataClassB import the following.
Name
DataInputs/DataClassA
DataInputs/DataClassB
sample-1
data-parent1,data-parent2
data-parent3
Data Classes can be linked to one another by parentage lineage using the same syntax. For example, a parent protein may produce many children proteins by some bio-engineering process. Use Data Classes to capture the parent protein and the children proteins.
Name
DataInputs/DataClassA
DataInputs/DataClassB
protein-1
data-parent1,data-parent2
data-parent3
protein-2
protein-1
protein-3
protein-1
protein-4
protein-1
Learn more about Sample Type and Data Class parentage relationships in this topic:
Choose one of these ways to assign names to the members of your Data Class:
Include a Name column in your uploaded data to provide a unique name for each row.
Provide a Naming Pattern when a Data Class is created. This will generate names for you.
Naming Patterns
The naming pattern can be concatenated from (1) fixed strings, (2) an auto-incrementing integer indicated by ${genid}, and (3) values from other columns in the Data Class. The following example is concatenated from three parts: "FOO" (a fixed string value), "${genid}" (an auto-incrementing integer), and ${barcode} (the value from the barcode column).
FOO-${genid}-${barcode}
Use naming patterns to generate a descriptive id, guaranteed to be unique, for the row.Learn more about naming patterns in this topic: Sample Naming Patterns
Aliases
You can specify alias names for members of a Data Class. On import, you can select one or more alias names. These aliases are intended to be used as "friendly" names, or tags; they aren't intended to be an alternative set of unique names. You can import a set of available aliases only via the client API. No graphical UI is currently available.
For Data Class members (such as bioregistry entities) with data dependencies or references, deletion is disallowed. This ensures integrity of your data in that the origins of the data will be retained for future reference. Data Class members cannot be deleted if they:
Are 'parents' of derived samples or other entities
Data Classes are used to represent virtual entity definitions such as Protein Sequences, Nucleotide Sequences, Cells Lines, and Expression Systems. See LabKey Data Structures. This topic describes creation and population of a Data Class within the user interface.
Add the web part "Data Classes" to see a list of those defined in the current folder, parent project, or /Shared project.
Enter > Page Admin Mode.
Select Data Classes from the selector in the lower left, then click Add.
Note that there is a "narrow" web part for Data Classes available on the the right side of the page. It does not have the controls described below, but can be used as a shortcut once you have set up your classes.
Click Exit Admin Mode.
You can create a Data Class from scratch, using the designer interface below, or by first defining a domain template, then using it to create similar classes.
Create New Data Class
To create a new Data Class, navigate to the container where you want it defined, then click New Data Class. If any templates are defined, this link will open a menu from which you'll select Design Manually to follow these steps. Creating a Data Class from a template is described below.
Data Class Properties
The first panel defines Data Class Properties.
Name: Required. This is the name of the Data Class itself.
The name must be unique, must start with a letter or number character, and cannot contain special characters or some reserved substrings listed here.
Description: Optional short description.
Naming Pattern: Unless your data will already include unique identifiers for each member of this class, use the Naming Pattern to specify how to generate unique names. Names can be generated with a combination of fixed strings, auto-incrementing integers, and column values. If you don't provide a naming pattern, your data must contain a Name column. Learn more about naming in this topic.
Category: Select a category, if applicable. For example, registry, media, and sources are categories of Data Class.
Sample Type: (Optional) The default sample type where new samples will be created for this Data Class.
Before saving, continue to add any Fields.
Fields
Click the Fields panel to open it.
Default System Fields
The top section shows Default System Fields that are always created with a Data Class.
Name: This field uniquely identifies each member of the Data Class. It is always required and cannot be disabled.
Description: This field can be disabled by unchecking the Enabled box, or required by checking the Required checkbox.
Click the to collapse this section and move on to any Custom Fields.
Custom Fields
To add custom fields, you have several options. When there are no fields defined yet you can:
Import or Infer Fields from File:
Infer them from a example data-bearing spreadsheet by dropping it into the upload area or clicking to browse and select it.
Supported file types are listed below the panel.
If your spreadsheet contains data for 'reserved fields', they will not be shown in the inferral, but are always created for you by the system. A banner will indicate that these fields are not shown.
Click Manually Define Fields, then use the Field Editor to add fields and set their properties. For details, see Field Editor.
Once you have defined the fields, click Save to create the new Data Class. You will see it listed.
Create Data Class from Template
If there is a Data Class template available, you can use it to create a new Data Class.Select New Data Class > Create From Template.Select the template to use and click Create.You will use the same Data Class designer described above to define the properties and fields for your new Data Class. The information from the template you selected will be prepopulated for you, and you can make adjustments as needed. Click Save to create the Data Class.
Populate a Data Class
To add data to a Data Class, you can insert individual rows or more commonly bulk import data from a spreadsheet.
In the Data Classes web part, click the name of the class you want to add data to.
In the grid below the properties, select (Insert data) and whichever import method you prefer:
Insert new row: Use a data entry panel with entry boxes for every field to add a single row.
Both import options allow you to Import Lookups by Alternate Key if you have lookups that should be resolved using a column other than the primary key.
Update or Merge Data During Import
Once your Data Class has been populated and is in use, new data can be added, and existing data can be updated in most cases. Select Import Options:
Add data(Default): Only add new data; if data for existing rows is included, the import will fail.
Update data: Update existing rows. All imported rows must match an existing row or the update will fail unless the following checkbox is checked.
Check the box to Allow new data during update to merge both updates to existing rows and addition of new rows.
When you update existing data, only the columns you update (for the rows you specify) will be changed. Existing data in other columns remains unchanged. This allows you to make a known change (update a review status for example) without needing to know the contents of all other columns.
For large-scale studies and research projects, success depends on:
Integrating many different kinds of information, including clinical, experimental, and sample data
Tracking progress of the protocol and associated research projects
Presenting your results in a secure way to colleagues
LabKey Study tools are designed around the specific requirements of longitudinal and cohort study management. Nearly every aspect of a study can be customized, including what data is captured by the system, the reports generated, and how the results are presented.To get started, we provide two example tutorials:
The default name for the individuals being studied is "participant". You can choose an alternate word to better match your working environment, such as "subject" or "patient", or can name the organism being studied, such as, "mouse", "mosquito", "tree", etc.ParticipantIDs must be no longer than 32 characters.
Time
Tracking how various attributes of your participants vary over time, or more generally, what events happen in what sequence, is a typical requirement of study research. LabKey Server provides three different ways of measuring time in your study.
Dates means that the time is broken into periods, called timepoints, bounded by calendar date, meaning that the amount of time elapsed between events is significant. The size of the timepoints can be customized so that months, weeks, years, or any other size 'blocks' become the units of time measurement.
Assigned Visits means that the data is divided into named "events" in a sequence, possibly but not necessarily corresponding to an individual visiting a location. The actual dates may not be relevant, only the sequence in which they occur. For instance, "enrollment", "first vaccination", "second screening" might be named visits in a study. In a visit based study, data collection events are assigned a "sequence number", possibly but not necessarily using date information provided.
Continuous is intended for open-ended observational studies that have no determinate end date, and no strong concept of dividing time into fixed periods. This style is useful for legacy or electronic health record (EHR) data.
Data
LabKey's study tools support the wide diversity of research data. Different data formats, methods of collection, analysis, and integration are brought together with convenient tools to support a wide variety of research efforts.The datasets in a study repository come in three different types:
Demographic.
One row per participant.
Demographic datasets record permanent characteristics of the participants which are collected only once for a study. Characteristics like birthplace, birth date, and enrollment date will not change over time.
From a database point of view, demographic datasets have one primary key: the participantId.
Each study needs at least one demographic dataset identifying the participants in a study.
Clinical.
One row per participant/timepoint pair.
Clinical datasets record participant characteristics that vary over time in the study, such as physical exam data and simple lab test data. Typical data includes weight, blood pressure, or lymphocyte counts. This data is collected at multiple times over the course of the study.
From a database point of view, clinical datasets have two primary keys: the participantId and a timepoint.
Assay/Instrument.
Multiple rows per participant/timepoint are allowed.
These datasets record the assay or instrument data in the study. Not only is this data typically collected repeatedly over time, but more than one of each per timepoint is possible, if, for example, multiple vials of blood are tested, or multiple dilutions of a product are tested on one sample.
From a database point of view, assay datasets have at least two, and possibly three keys: participant ID and timepoint, plus an optional third key such as a sampleID.
Integrate
LabKey Server knows how to integrate the diverse range of research data, including demographic, clinical, experimental, and sample data.
Track Progress
Progress tracking tools help you put all the pieces together.
Present Results
Present your research using LabKey's rich analytic and visualization tools.
Learn More
Use the following topics to get started:
Tutorial: Longitudinal Studies - Learn how to use LabKey's tools for observational studies by taking a guided tour of an example research study.
This tutorial provides a high-level walkthrough of LabKey longitudinal research studies. It focuses on data collection, quality control, and dashboards and reports available. The walkthrough uses a publicly available online example study.The online example provides a (fictional) research effort examining how HIV affects the immune system. The researchers have completed these phases of the project:
Collected basic demographic and physiological data from the study participants.
Performed a number of blood tests and assays on the participants over time.
Imported and configured the clinical, mechanistic, and other data into a study in LabKey Server.
Created custom built reports to analyze the results.
This walkthrough tutorial provides a tour of this (fictional) research effort so far. The walkthrough is divided into three steps:
This tutorial provides a walkthrough of LabKey's "longitudinal study" features, which help researchers to curate their datasets, analyze them, and publish the results. The study toolkit also includes quality control features, an intuitive visualization builder, data de-identification, and much more.
Note that all of the links in this tutorial open their targets in new browser tabs. To open the links in a new window, hold down the Shift key and click the link.
In this first step we look at the main dashboards of the study. Each tab provides different information and tools organized by what users need. Note that all data in this example study is fictional and created by random data generators.
The LabKey Studies tab includes some context about how LabKey studies help researchers manage clinical data.
Overview Tab
Click the Overview tab to open the main dashboard for the (simulated) research effort.
Here you see a summary of the study contents, a navigator section, details about the funding grant and investigator, and other contextual information an administrator has provided.
This tab displays the base datasets and reports built on top of the base datasets, such as joined data grids and visualizations. Hover over a link to see a popup preview of the item.
In this step we will look at the variety of reports available in the online example. We will look at both built-in reports (automatically available in all study folders) and custom reports (created by the study researchers or data analysts).As you progress with the walkthrough, notice two things:
First, all of these reports are 'live', that is, they represent the current state of the underlying data. If the data were changed, all of these reports would be updated without any user interaction.
Second, most of these reports support viewer customization "on the fly", such as by filtering the underlying data, or changing report parameters. This empowers your (authorized) audience to apply their own questions to the research data as they explore.
Note that all of the links in this tutorial open their targets in new browser tabs. To open the links in a new window, hold down the Shift key and click the link.
View Built-in Reports
The following built-in reports are automatically part of a LabKey study folder. Note that this tour touches on only a subset of the available built-in reports.
This overview report displays a high-level view of data collection in the study and provides a convenient jumping-off point to specific facets of the data.
Each dataset is listed as a row. Each timepoint or visit is listed as a column. The cells indicate how much data is available for each dataset for that particular time in the study.
Each cell's displayed number is a link to a filtered view of that data.
The Study Overview also shows whether participants are completing visits and can help a researcher confirm that the necessary data is being gathered for the purpose of the study. Seen below, with physical exam data collected for 12 participants per visit, the study seems on track.
Notice that the dropdown options above the grid allow you to filter by either Participant's current cohort or QC State of the data. The following sections explore these options.
You can also use the checkboxes to switch between viewing counts of participants and counts of rows for each visit.
Quality Control: Study-wide Picture of QC State
You can filter the Study Overview grid to provide a view of the quality control states of data across the whole study.
The Overview is also filterable by quality control state. As shown below, the report has been modified to show only data that has 'not been reviewed' by the quality control process. Use the dropdown to see other quality control states.
Quality Control: Not Yet Reviewed Details
If you click the number in a cell in the overview report, you'll go directly to the details of which data is represented by that count. You can also find rows needing attention from within a dataset, such as by using conditional formatting on the QC State column, as shown in the custom grid view in this report.
OR, to navigate manually: Click the Overview tab, then click the Study Navigator, then click the link at the intersection of the Physical Exam row and the Visit 2.0 column.
The resulting data grid shows all physical exams conducted at "visit 2" for each participant.
Also notice that the Systolic Blood Pressure column makes use of conditional formatting to display values configured to be noteworthy in blue (low) and red (high).
OR, to navigate manually, click PT-105 from the Participants tab.
The participant page shows demographic data and a list of datasets isolated out for this participant, PT-105.
Click the (collapse) icon for the MedicalHistory dataset.
Click the (expand) icon next to 5004: Lab Results to open details of this dataset for this participant.
If you access data for an individual participant from within a dataset where participants are ordered, you will have additional links for scrolling through the different subjects: Next Participant and Previous Participant. These are not provided if you directly access the participant page.
Custom Reports
The following reports were custom built by administrators and data specialists.
This data grid is combined from 4 different base datasets: Demographics, Lab Results, ViralLoad-PRC, and ViralLoad-NASBA.
All study datasets share the Participant ID, Visit, and Date columns.
When you combine datasets in this manner, aligning by participant over time, you unlock the ability to analyze information that spans multiple spreadsheets easily. All the built in tools for creating plots and reports can be applied to these 'joined' dataset grids.
Visualizations can communicate study findings quickly and clearly. Rather than expecting reviewers to immediately understand results from grids of data, you can present summary plots showing relationships and trends that are meaningful at a glance. In this step we will generate our own charts and export them. We will make 2 charts:
Chart 1 will show CD4 counts for different individuals over time
Chart 2 will show how the mean CD4 counts vary for different groups/cohorts over time
Create a Time Chart
Navigate to the Lab Results dataset. (Opens in a new tab.)
On the data grid, select (Charts/Reports) > Create Chart.
In the Create a plot dialog, click Time on the left.
In the X-Axis panel, select Visit-based.
Drag CD4 from the column list on the right into the Y Axis box.
Click Apply.
You will see a time chart like the following image. By default, the first 5 participants are shown.
In the left hand Filters panel, you can select individual participants to include in the chart. Clicking any value label will select only the given participant, checking/unchecking the box allows you to toggle participants on and off the chart.
Visualize Performance of Participant Groups
In this step we will modify the chart to compare the CD4 levels for different treatment groups.
Click Chart Layout in the upper right.
Under Subject Selection, choose Participant Groups.
Click Apply.
You now have a chart comparing the mean CD4+ levels of the different treatment groups over time.
The filters panel now shows checkboxes for participant groups defined in the study.
Only groups checked will be shown in the chart allowing you to filter the time chart in various useful ways.
Hover in the upper-left for export options. You can export the chart image in PDF or PNG formats.
Congratulations
You have now completed the walkthrough of LabKey tools for longitudinal studies. Next, you can explore creating a study from scratch in this tutorial:
This tutorial teaches you how import longitudinal research data into LabKey Server. Using example data files, you will assemble and configure the core datasets, perform analyses, create visualizations, and learn about cohorts and participants along the way.The aligned data provides a unified picture of the study results that can be more easily analyzed as a "whole" than disparate parts.All example data used is fictional.
In this step, you will create a LabKey Study folder and set some basic study properties.
Create a Study Folder
Log in to your server and navigate to your "Tutorials" project. Create it if necessary.
If you don't already have a server where you can create projects, start here.
If you don't know how to create projects and folders, review this topic.
Create a new subfolder named "Study Tutorial". Choose folder type "Study" and accept other defaults.
On the home page of the new study, click Create Study.
Set Study Properties
Study properties are where you store some basic information about your study, including a name, identifying information, etc. You may also customize the word used to describe individual participants in a study (participants, subjects, mice, etc.) Learn more about study properties in this topic: Study Properties.For this tutorial, we use the default study properties, described below:
Look and Feel Properties
Study Label - This label is displayed in the main banner. Customize this label as desired.
Subject Noun (Singular) & (Plural) - These nouns are used throughout the user interface. Other possible values: "Subject", "Primate", "Mouse", etc.
Subject Column Name - When importing data, LabKey assumes this column contains the primary id/key values.
Visit/Timepoint Tracking
Timepoint Style - This choice depends on the nature of the data being imported. "Dates" means that the study timepoints are imported as calendar dates. "Assigned Visits" means that the timepoints are imported as a sequence of integers or decimal numbers. Use "Continuous" if your data has calendar dates and the observational window has no set beginning or end dates. This choice cannot be modified after the study has been initially created.
Start Date - Used in Date-based studies as the default first date of observation. This value is ignored in Visit-based and Continuous studies.
Security
Security Mode - Four different security styles are available.
On the Create Study dialog, leave all the default values in place.
Note that if you would like to create additional custom study properties, you can do so at the project level. Learn more in: Study Properties.
Scroll to the bottom of the page and click Create Study.
After creating the study, you will initially land on the Manage tab, the administrator dashboard for managing many study features.
An easy way to create a new dataset is to use a data file and infer an initial set of field names and data types to create the dataset structure and importing the initial set of data simultaneously.In this step, we will import four datasets:
Demographics - This dataset describes the participants in the study.
Physical Exam - A record of vital signs (pulse, blood pressure, etc.) measured over the course of the study.
Lab Results - This dataset contains various white blood cell counts.
Viral Load PCR - Contains the viral counts measured using a polymerase chain reaction method.
Because this is a "visit-based" study, LabKey will align these datasets based on the ParticipantId and SequenceNum fields. (If it were a "date-based" study, it would align the datasets based on the ParticipantId and Date fields.) This alignment makes reporting and analysis of the data much easier, as we will see in step 4 of the tutorial.
In this step, we create a dataset based on the Demographics.xls spreadsheet data.
Click the Manage tab.
Click Manage Datasets.
Click Create New Dataset.
On the Create Dataset Definition page, first define Dataset Properties:
Name: Enter "Demographics"
Under Data Row Uniqueness, select Participants only (demographic data).
Click the Fields section to open it.
Drag the Demographics.xls file you downloaded into the Import or infer fields from file target area.
You will see the fields inferred.
If necessary, you could make changes here. The exception is that key fields, in this case "ParticipantId" are "Locked" and cannot be changed.
Learn about editing and adding new fields in this topic: Field Editor.
Review the field names and data types that were inferred, but make no changes for this tutorial.
Scroll down to see the Column Mapping section.
Here the server has made a best guess at the fields that will map to your keys.
In this case, the inferral for participant information is correct: the "ParticipantId" column.
For the Visits dropdown, select SequenceNum. Notice this locks the "SequenceNum" field.
Below this, the Import data from this file upon dataset creation? section is set up by default to import the data as the dataset is created. You see a preview of the first few lines. Make no changes for this tutorial.
Click Save to create and populate the dataset.
Click View Data and you will see this dataset.
Import PhysicalExam.xls
Repeat the import process above for the PhysicalExam.xls file, with one difference:
Under Data Row Uniqueness, leave the default selection Participants and Visits.
Continue to infer fields, set mappings, and import the data.
Import LabResults.xls
Repeat the import process above for the LabResults.xls file.
Import ViralLoadPCR.xls
Repeat the import process above for the ViralLoadPCR.xls file.
Confirm Data Import
To confirm the data import, click the following tabs, and see how the presence of data has changed the dashboards on each:
Overview tab, click the Study Navigator.
Participants tab, click a participant ID.
Clinical and Assay Data tab, see your datasets listed.
Cohorts are be used to group participants by characteristics, such as disease state, for efficient analysis.
It this step we will assign participants to cohorts based on "treatment group". Four cohort groups will be used:
DRV - Participants who have received the DRV treatment regimen
ETR - Participants who have received the ETR treatment regimen
HIV Negative - Participants who are HIV negative and have received no treatment
No Treatment - Participants who are HIV positive but have received no treatment
Once the cohorts have been established we will be able to compare disease progression in these groups.
Assign Cohorts
We will assign cohorts automatically based on the "Treatment Group" field in the Demographics dataset.
Click the Participants tab and notice that all participants are "Not in any cohort", giving no way to filter them here.
Go to the Manage tab and click Manage Cohorts.
If necessary, select Automatic Participant/Cohort Assignment
Select the "Demographics" dataset from the Participant/Cohort Dataset dropdown menu.
For the Cohort Field Name, select Treatment Group.
Click Update Assignments.
You will see the list of Defined Cohorts populate with the two existing values in the "Group Assignments" column, followed by a list showing which participants are assigned to which cohort. Any participants without a group assignment will not be assigned to a cohort.
View Cohorts
Now return to the Participants tab and notice that the new cohorts you just created. Hover over a cohort value to show which participants are included in that option: excluded participant ids are grayed out, so you can get a quick visual indication of the prevalence of that option. Click a label or use checkboxes to filter the list displayed.You can also type to filter using the entry box above the participant set.
Now that your datasets have been aligned with participant and time information, and cohort information has been extracted, you can start to take advantage of the integrated data. In this step you will create grids and visualizations based on the aligned data.
Visualize Cohort Performance
LabKey makes it easy to incorporate cohort categories into visualizations. To see this, we will create a visualization that compares blood cell counts across the cohort groups.
Click the Clinical and Assay Data tab and click the Lab Results dataset.
Click the header for the CD4 column and select Quick Chart.
LabKey Server will make a "best guess" visualization of the column. In this case, a box plot which shows the distribution of values, organized by cohort groups:
Notice that you are now in the main plot editor and could change the chart type and layout as you like.
Optionally, you can click Save, name the plot "CD4 Counts by Cohort", and click Save.
If you save this chart, click the Clinical and Assay Data tab. Notice that the chart has been added to the list.
Create Data Grids from Combined Datasets
What if you want to compare two measurements that are in different datasets? For example, is there a relationship between CD4 cell counts (from the "Lab Results" spreadsheet) the blood pressure measurements (from the "Physical Exam" spreadsheet)? Now that the spreadsheets have been imported to the LabKey study as datasets, we can easily create a combined, or joined view, of the two.
Go to the Clinical and Assay Data tab.
Navigate to one of the datasets to be combined. In this case, click the Lab Results dataset.
Select (Grid Views) > Customize Grid.
In the Available Fields panel, scroll down and open the node DataSets by clicking the . (Notice that DataSets is greyed out, as are the names of the datasets themselves. This only means that these are not "fields" or columns to select for display, but nodes you can open in order to select the columns they contain.)
Open the node Physical Exam.
Check the boxes for Systolic Blood Pressure and Diastolic Blood Pressure. This indicates that you wish to add these columns to the grid you are viewing. Notice they are added to the list in the Selected Fields panel.
Click Save.
In the Save Custom Grid View dialog, select Named and enter "Lab Results and Physical Exam Merged". Click Save.
You now have a joined data grid containing columns from both the Lab Results and Physical Exam datasets. Notice the view name in the header bar (inside the red oval). Also notice that this new grid is added to the Clinical and Assay Data tab.
Plot Combined Data
Confirm you are viewing the combined grid view named "Lab Results and Physical Exam Merged".
Select (Charts) > Create Chart. This opens the chart designer where you can create and customize many types of charts. The Bar plot type is selected by default.
In the left panel, click Scatter.
For the X Axis, select CD4 by dragging it from the list of columns on the right into the box. This column came from the "Lab Results" dataset.
For the Y axis, select Systolic Blood Pressure. This column came from the "Physical Exam" dataset.
Click Apply.
Click Save, name the report "CD4+ Counts vs. Blood Pressure", and click Save in the popup.
Your chart will now be added to the Clinical and Assay Data tab.
Plot Trends Over Time
A key part of longitudinal study research is identifying trends in data over time. In this example, we will create a chart that shows cell counts for the cohorts over the course of the study.
Click the Clinical and Assay Data tab.
Click the Lab Results dataset.
Select (Charts) > Create Chart.
Click Time.
Notice there are no columns to plot.
Time charts require that columns used have been explicitly marked as "measures". To do so we edit the dataset definition:
Click Cancel to leave the plot editor and return to the grid for Lab Results.
Click Manage in the header bar of the grid, then click Edit Definition.
Click the Fields section to open it.
Expand the CD4 field.
Click Advanced Settings.
In the popup, check the box for Make this field available as a measure.
To increase your charting options, repeat for the other 2 numeric fields (Lymphocytes and Hemoglobin).
Click Apply, then Save.
Now we are ready to plot cell counts on a time chart.
Click View Data to return to the "Lab Results" data grid.
Select (Charts) > Create Chart.
Click Time again. Notice now that the columns you marked as measures are listed on the right.
Choose Visit-based for the X Axis field.
Drag the CD4 column to the Y Axis box.
Click Apply.
By default, the time chart plots for the first 5 individual participants in the data grid. To instead compare trends in the cohorts:
Click Chart Layout where you can customize the look and feel of a chart.
Change the Subject Selection to Participant Groups.
Click Apply.
There are now four lines, one for each cohort. Now we see a clear trend (in our fictional data) where CD4 levels in the HIV negative cohort generally stay flat over time, while the other cohorts vary over the 24 months studied.
Click Save.
Name your time chart "CD4 Trends" and click Save in the popup.
Congratulations
You have completed the tutorial! See below for related topics and tutorials.
Next Steps
This tutorial has given you a quick tour of LabKey Server's data integration features. There are many features that we haven't looked at, including:
Custom participant groups. You can create groups that are independent of the cohorts and compare these groups against one another. For example, it would be natural to compare those receiving ARV treatment against those not receiving ARV treatment. For details see Participant Groups.
Assay data. We have yet to add any assay data to our study. For details on adding assay data to a study, see Tutorial: Import Experimental / Assay Data. You can treat this assay tutorial as a continuation of this tutorial: simply use the same study repository and continue on to the assay tutorial.
R reports. You can build R reports directly on top of the integrated study data. For details see R Reports.
Navigate to a project where (1) you have administrator permissions and (2) where it is appropriate to upload test data. Your own "Tutorials" project is one option.
In that project, create a folder named "HIV Study" of type "study".
On the New Study page, click Import Study.
Confirm Local zip archive is selected, then click Choose File. Navigate to and select the "ExampleResearchStudy.folder.zip" file you downloaded.
Click Import Study.
Wait for LabKey Server to process the file: you will see a status notification of Complete when it is finished. You might see a status of Import Folder Waiting on a busy or shared server. If the page doesn't update automatically after a few minutes, refresh your browser window.
Click the HIV Study link near the top of the page to go to the home page of your new study.
Next Steps
Now you have an example study that you can explore and modify. See the following topics for more you can try:
The following topics explain how LabKey studies work from a study user's perspective, not an administrator's perspective. They explain how to navigate data, work with cohorts, import data, and ensure data quality control -- in short, the "day to day" operations of an up and running LabKey study.
A study folder contains a set of tools for working with data as part of a cohort, observational, or longitudinal study where subjects are tracked over time. This topic covers the basics of navigating a study.
The Study Folder provides the jumping-off point for working with datasets in a Study.By default, the main study folder page includes several tabs. On narrower browsers, the tabs will be collapsed as a single pulldown menu.
The Overview tab displays the title, abstract, a link to the protocol document, and other basic info. Including a link to the study navigator.
The Participants tab shows the demographics of the study subjects.
The Clinical and Assay Data tab shows important visualizations and a data browser to all study data. Clicking on a dataset brings you to the dataset's grid view. Also all of the views and reports are listed.
The Study Navigator provides a calendar-based view of Study datasets. To view it, click Study Navigator under the grid graphic. All datasets are shown as rows, timepoints or visits as columns.
Use pulldowns to optionally filter to see data for a specific cohort or quality control status.
Checkboxes control whether the grid shows the participant count or number of data rows for the given dataset during that timepoint. When both are checked, counts are separated by a slash with the participant count first.
The leftmost column of counts aggregates the counts for all timepoints.
View Data by Cohort
Use the Participant's current cohort to show only data for one specific cohort at a time, or select All to see all participants. Note that when viewing a single cohort, if any visit (or timepoint) is associated with a particular cohort, it will only be shown in the study navigator when that cohort (or "All" cohorts) is selected.
View Data by Timepoint or Visit
To display information collected for a particular dataset at a particular visit or timepoint, click the number at the intersection of the dataset and visit you are interested in to see a filtered view of that dataset. For example, click on the number at the intersection of the "Visit 2" column and the Physical Exam row. The resulting data grid view shows all physical exams conducted on the second visit in the study for each participant.You can see an interactive example here.
Hover for Visit Descriptions
If you have entered Description information for your timepoints or visits, you will see a question mark next to the timepoint name (or number). Hover over the '?' for a tooltip showing that description information.See Edit Visits or Timepoints for more information.
A cohort is a group of participants who share a particular demographic or study characteristic (e.g., all the participants on one treatment, or all those with a particular HIV status). Cohorts can be either:
Simple: Assignments are fixed throughout the study.
Advanced: Assignments can change over time depending on some study characteristic, such as disease progression or treatment response.
Cohorts are similar to participant groups, but have a some additional built in behavior. You can only have one set of cohorts in a study, but multiple kinds of participant groups as needed. An administrator must define and manage study cohorts; editors can add participant groups.
View Cohort Assignments
Within a dataset, you can see cohort assignments in a grid that includes the Cohort column. For example, in the interactive example demo study, the "Demographics" dataset shows which cohort each participant is in.If you don't see this column, you can add it to any study dataset following the steps listed below.
Filter Data by Cohort
When the "Cohort" column is shown in the grid, the dataset can be filtered directly from the column header. In addition, study datasets which don't show this column can still be filtered by cohort using the Groups menu, Cohorts option:
Changing Cohorts Over Time
If your administrator has set up "Advanced" cohorts, cohort assignments can vary over time. If this is the case, your Groups > Cohorts menu will contain additional options. You will be able to filter by cohort and select one of the following:
Initial cohort: All participants that began the study in the selected cohort.
Current cohort: All participants that currently belong to the selected cohort.
Cohort as of data collection: Participants who belonged to the selected cohort at the time when the data or a sample was obtained.
View Individual Participants by Cohort
You can step through per-participant details exclusively for a particular cohort.
Display a dataset of interest.
Filter it by the desired cohort using Groups > Cohorts.
Click a participantID.
You now see a per-participant details view for this member of the cohort. Click "Next Participant" or "Previous Participant" to step through participants who are included in this cohort.
Customize Grid to Show Cohorts Column
To add the "Cohort" column in any dataset in the study, make use of the implicit join of study data using the ParticipantID.
Display the dataset of interest.
Select (Grid Views) > Customize Grid.
In the left panel, expand the ParticipantID node.
Check the box for Cohort.
In the Selected Fields panel, you can reorder the fields by dragging and dropping.
Click Save and name your custom grid, or make it the default for the page.
Participant groups provide an easy way to group together study subjects, either for scientific/analytical reasons or utilitarian/management reasons.
Highlight a scientifically interesting group of participants, such as participants who have a certain condition, medical history, or demographic property.
Mark off a useful group from a study management point of view, such as participants who have granted testing consent, or participants who visited such-and-such a lab during such-and-such a time period.
Cohorts are similar to participant groups, with some additional built in behavior. You can only have one set of cohorts in a study, but multiple participant groups and categories. An administrator must define study cohorts; non-admins can add participant groups, as described below.
Participant groups are organized into categories. Within each participant group category, each participant can only appear once. This allows for "cohort-like" segmenting of all the participants based on a given set of criteria (i.e. with and without some characteristic) as well as "singleton" groups, where a subset of participants is of interest, but the subset of "not that group" does not need to be maintained.
You can create a participant group by individually selecting participants from any data grid in your study.
Go to any data grid in your study that includes a ParticipantID column.
Individually select the participants you want to include in the group using the checkbox selectors.
Select Groups > Create Participant Group > From Selected Participants.
Enter the Participant Group Label
Review the set of Participant Identifiers. Edit if necessary.
Select a Participant Category from the pulldown or type in the box to define a new category. Note that participants can only be in one group within any given category. If you leave the category field blank, the new group will be listed on the top level menu as its own "category".
Use the checkbox if you want to share the category you define with others.
Click Save.
Create a Participant Group by Filtering
You can create a participant group by filtering a grid to show only the participants you want to include.
Note that participant groups are static. Meaning that this filtering applies only to the process of creating the group of participants initially. The filter will not be applied later to add new participants that would have met the criteria if they had been present at group creation.In addition, the filter will not be applied when you view data for those participants. The group is composed of the 'filtered' subset of participants, but when you view the group in that dataset, you may see many more rows if the same participants had other values in that column at other times.
Go to any data grid in your study that includes a ParticipantID column.
Filter the data grid to include only the participants you want to include in the group.
Use the checkbox to select the displayed rows.
Select Groups > Create Participant Group > From Selected Participants.
Enter the Label and Category, and review the list as described above.
Click Save.
A common process is to then add additional groups in this category using different filter values. For example, if you had a boolean column, you might first create a group for the "true" value in that column, then create a second group in the same category for the "false" value in the same column.
Manage Participant Groups
From any study grid, select Groups > Manage Participant Groups.
Click Create to create a new participant group directly from this page. This creation dialog includes a grid section for specifying the group:
Use the dropdown Select Participants from to select the data grid to show.
Select individuals using the checkboxes, then click Add Selected, or filter the grid using the column headers to show the group to create, then click Add All.
Choose or name the Participant Category and elect whether to share it.
Click Save.
Click any existing group to select it and enable options:
Delete Selected
Edit Selected: Opens the same creation dialog for editing, including the selection grid.
Share Groups and Categories
When creating groups, keep in mind:
Shared/Not Shared applies directly to the category and only indirectly to the participant groups within a category.
Administrators and editors can create shared or private groups; everyone with read access can create private groups.
Admins and editors can delete shared groups, otherwise you have to be the owner to delete a group.
Anyone with read access to a folder can see shared groups.
Use a Participant Group
Once a participant group has been created you can filter any of the datasets using that group, just as you can filter using a cohort.
Open the Groups menu for a list of available cohort-based and group-based filters, including any groups you have created or have been shared with you. Any group that is not in a category with other groups is shown on the main menu (as a "singleton" group); categories with more than one group become submenus.
Delete a Participant Group
Open the Manage tab.
Select Manage Participant Groups.
Highlight the participant group you wish to delete, then click Delete Selected.
Dataset quality control (QC) states allow you to track the extent of quality control that has been performed on study data. Administrators define the available QC states, often to reflect an approval workflow, such as:
Not Yet Reviewed
Passed Review
Did Not Pass Review
Quality control states can be assigned to datasets as a whole or on a row-by-row basis.LabKey's dataset QC features allow you to mark incoming datasets with the level of quality control executed before incorporation of the dataset into a study. Once datasets are part of a study, the QC markers for individual rows or entire datasets can be defined to recognize subsequent rounds of quality control.
The quality control process allows study administrators to define a series of approval and review states for data.
Each can be associated with "public" or "nonpublic" settings that define the initial/default visibility of the data, though do not permanently limit visibility.
Different QC states can be assigned to incoming data depending on the import method used. A default QC states can be assigned independently for:
(1) datasets imported via the pipeline
(2) data linked to the study from an assay or sample type
(3) directly inserted/updated dataset data
Reviewers can filter a data grid by QC State and thus find all data requiring review from a single screen.
All quality control actions are audited in the audit log.
Note that the "public/nonpublic" setting is a convenience but not a security feature, because it does not use Security roles to determine visibility. These settings only control how the data is initially displayed/filtered when the dataset grid is rendered. Users with the Reader role in the study, are able to re-filter the grid to see all data regardless of QC states.In contrast, assay QC states can be configured to utilize security roles to determine data visibility in a more secure way.
View QC States Study-Wide
Each study comes with a built-in report that shows QC states across all datasets.
To view the report:
Click the Overview tab.
Click Study Navigator.
Use the QC State dropdown to control the report.
The numbers in the grid show the number of participants and data rows in the selected QC state. Click these numbers to view details.
Filter Individual Datasets by QC State
Users can filter data grids to display a particular quality control state. Reports, statistics, and visualization created from such filtered grids will respect the QC state filtering.
Go to the dataset of interest.
Click QC States. If the "QC States" button is not visible above your dataset, ask your admin to set up dataset-level QC.
Choose one of the options listed. All defined states are included, plus All Data, Public/approved Data, and Private/non-approved data.
For example, the following screenshot shows how you would select all rows of data that remain in the "Not Yet Reviewed" QC state.
Change the QC State
You can change the QC state for dataset rows if you have sufficient permission (i.e., the Editor role or greater).
The topics in this section help study administrators set up, integrate, and manage study data. This page also outlines the study data model, helpful for planning study structure.
The core entities of a Study (its "keys") are Participants (identified by "Participant IDs") and Time, either Visits (identified by "Visit IDs" or "SequenceNums") or Dates.Participants appear at planned locations (Sites) at expected points in time (Visits) for data collection. At such Visits, scientific personnel collect Datasets (including Clinical and Assay Datasets) and may also collect specimen samples. This data can all be uploaded or linked to the Study.Learn more about managing samples in a study in this topic:
The default set of study properties provide basic information including a name, description, how to identify study subjects and track time, and the option to attach protocol documents. Administrators set some of these properties during study creation, and can edit all the study properties from the Manage tab.Additional custom study properties can be added to the definition of a study at the project level, allowing you to store and use information about multiple studies in a given project beyond the built in study properties. Additional properties are exported and imported as part of folder archives.
Note that while study properties are defined from within a study folder, they will be created at the project level and thus defined within every study folder in that project. Within each folder, the value of the property can be set. This supports a consistent schema for cross-study queries and reports within a project.Only a project administrator can add or delete custom study properties, and changes made to the properties available will be reflected in every study folder within the project.
See and set defined study properties either by clicking the pencil icon in the Overview web part or by clicking Edit Study Properties from the Manage tab.
Label: By default, the folder name is used with the word "Study" added.
Investigator: Enter a name to display in the overview.
Grant/Species: If you enter the grant name and/or species under study, they will be displayed. These fields also enable searches across many study folders to locate related research.
Description/Render Type: Our example shows some simple text. You can include as much or as little information here as you like. Select a different Render Type to use HTML, Markdown, Wiki syntax, or just plain text.
Protocol Documents: Click Attach a file to upload one or more documents. Links to download them will be included in the Study Overview web part.
Timepoint Type: How data is sequenced in the study. See Visits and Dates to learn more.
Start/End Date: See and change the timeframe of your study if necessary. Note that participants can also have individual start dates. Changing the start date for a study in progress should be done with caution.
Subject Noun: By default, study subjects are called "participants" but you can change that here to "subject," "mouse," "yeast," or whatever noun you choose.
Subject Column Name: Enter the name of the column in your datasets that contains IDs for your study subjects. You do not need to change this field to match the subject nouns you use.
Click Add Field and specify the Name and Data Type for each field required.
To use a Label other than the field name itself, expand the panel and enter the label text.
Reorder custom fields by dragging and dropping them using the six block "handle" on the left.
Delete unneeded fields by expanding the panel and clicking Remove Field.
Click Save when finished.
For example, you might add additional administrative properties like these:Once added you will be able to set values for them via the Study Properties link from the Manage tab. Custom properties appear at the end of the list after the built in study properties.
Administrators can create and manage datasets in a study following the pathways outlined in this topic. Begin on the Manage tab of a study and click Manage Datasets.
The schedule identifies expected datasets for each timepoint. Learn more in this topic: Study Schedule
Change Display Order
Datasets can be displayed in any order. The default is to display them ordered first by category, then by dataset ID within each category.To change their order, click Change Display Order, then select a dataset and click an option:
Move Up (one spot)
Move Down (one spot)
Move to Top
Move to Bottom
Click Save to save your new order, or Reset Order to return to the default ordering. Once confirmed, the reset order action cannot be undone.
Change Properties
Edit these properties of multiple datasets from one screen by clicking Change Properties:
Label
Category
Cohort
Status (one of None, Final, Draft, Locked, or Unlocked)
Visibility
After making changes, click Save. You can also Manage Categories from this page.To edit properties for an individual dataset instead, go back then click the dataset name from the Manage Datasets page instead. See below.
Delete Multiple Datasets
An administrator can select multiple datasets at once for deletion. The id, label, category, type, and number of data rows are shown to assist in the selection.After deleting datasets, it is advisable to recalculate visits/dates, in order to clear any empty visit and participant references that may remain in the study.
View Audit Events
Click this link to see the DatasetAuditEvent log, filtered to the current container.
Manage Individual Datasets
To manage a specific dataset, click the name in the Datasets panel of the Manage Datasets page. Administrators can also directly access this page by clicking Manage in the dataset header menu.See Dataset Properties.
Datasets Web Part
If you would like to display a simple directory of datasets by category, add a Datasets web part. Enter > Page Admin Mode, then select Datasets from the Select Web Part pulldown in the lower left. Click Add. Click Exit Admin Mode when finished.
Related Topics
Dataset Properties: Options for managing or deleting an individual dataset and viewing or adjusting its properties.
When you create a study, you specify how time will be tracked, i.e. whether the study is visit-based or date-based. Both are similar methods of separating data into sequential buckets for tracking data about participants over the events in a research study.This topic describes the options available for managing either visits or timepoints. You will be able to see which type of study you are using by what the name is on the link you click to reach the manage page.
Note that if you don't see either link, your study is defined as continuous, in which data is time-based but not managed in either timepoints or visits.
Change Visit Order: Change the display order and/or chronological order of visits.
Change Properties: The label, cohort, type, and default visibility of multiple visits can be edited at the same time.
Delete Multiple Visits: Select the visits you want to delete. Note this will also delete any related dataset rows. The number of rows associated with each visit are shown for reference.
Delete Unused Visits: If there are any visits not associated with any data, you can delete them. You will see a list of unused visits and confirm the deletion.
Recalculate Visit Dates: The number of rows updated is shown after recalculating.
Visit Import Mapping: Define a mapping between visit names and numbers so that data containing only visit names can be imported. Provide the mapping by pasting a tab-delimited file that includes Name and SequenceNum columns.
The table of Visits shown on the Manage Visits page displays some basic details about the currently defined set of visits.Note that the Label column displays either the Label field if set, or the Sequence if the label has not been set. Edit a row using the icon to see if the label has been set.
Change Visit Order
Display order determines the order in which visits appear in reports and views for all study data. By default, visits are displayed in order of increasing visit ID for visit-based studies, which is often, but not necessarily the same as date order as used in timepoint-based studies. You can also explicitly set the display order.Chronological visit order is used to determine which visits occurred before or after others. Visits are chronologically ordered when all participants move only downward through the visit list. Any given participant may skip some visits, depending on cohort assignment or other factors. It is generally not useful to set a chronological order for date-based studies.To explicitly set either order, check the box, then use the "Move Up" or "Move Down" buttons to adjust the order if needed. Click "Save" when you are done.
Recompute Timepoints: If you edit the day range of timepoints, use this link to assign dataset data to the correct timepoints. The number of rows changed will be reported.
Delete Multiple Timepoints: Select the timepoints you want to delete. Note this will also delete any related dataset rows. The number of rows associated with each timepoint are shown for reference.
Set the study start date and duration for timepoints. Used to assign a row to the correct timepoint when only a date field is provided.
A timepoint is assigned to each dataset row by computing the number of days between a subject's start date and the date supplied in the row.
Each subject can have an individual start date specified by providing a StartDate field in a demographic dataset. Note that if you do not immediately see these start dates used, you may have to click Recompute Timepoints on the Manage Timepoints page.
If no start date is available for a subject, the study start date specified here is used.
The default timepoint duration will determine the number of days included in any automatically created timepoints.
Disallow Automatic Visit or Timepoint Creation
By default, a new visit or timepoint will be created in the study during the insert or update of any dataset or specimen rows which do not already belong in an existing visit or timepoint. If you would like to disallow this auto creation and disallow new visits or timepoints, in the Automatic Visit (or Timepoint) Creation section, check the box to Fail data import for undefined visits.
The Study Schedule helps a study administrator track progress, identify requirements by timepoint or visit, and see at a glance how their data collection is proceeding.
From within your study, click the Manage tab, then click Study Schedule.You'll see the current status and requirements in a dashboard helping you track progress.
Set Required Datasets
To mark a dataset as required for a given timepoint or visit, click a cell within the grid, as shown below.
Edit Dataset Schedule Properties
Click the to edit properties for a dataset, including Author, Data Cut Date, Category, Description, and Visibility. You will also see here when the dataset was last modified.The Visibility setting controls how data is displayed in the Study Data Views web part.When you edit the Status of a dataset, options are: Draft, Final, Locked, Unlocked.
Note: Locked/Unlocked is not enforced by LabKey Server: setting a dataset to Locked does not prevent edits to the dataset.
In a LabKey Study, the places where collection and storage of material and information take place are collectively referred to as locations. Administrators can manage and change the study locations available, and assign one or more of the location types to each physical location. Location types include:
clinics
repositories
labs
Note that if you are using the specimen module, you will have additional specimen-related options. Learn more in the documentation archives.
Managing locations and changing location definitions requires folder or project administrator permissions. However, the contents of the table can be made visible to anyone with read permissions by adding a locations query web part.
Click the study's Manage tab.
Select Manage Locations.
Add a Location
Click (Insert New Row), enter information as required, then click Save. Fields include:
Location Id: The unique LDMS identifier assigned to each location, if imported from an LDMS system.
External Id: The external identifier for each location, if imported from an external data source.
Labware Lab Code: The unique Labware identifier assigned to each location, if imported from a Labware system.
Lab Upload Code: The upload code for each location.
Label(Required): The name of the location.
Description: A short description of each location.
Location Types: Check boxes to select which type of location this is (more than one may apply): SAL (Site-Affiliated Laboratory), Repository, Endpoint, Clinic.
Street Address/City/Governing District/Postal Area: Physical location.
Edit an Existing Location
Hover over a row and click (Edit) to change any information associated with an existing location.
Delete Unused Locations
Locations which are not in use may be deleted from the Manage Locations page, shown above. The In Use column reads true for locations referred to by other tables within the study.To delete specific locations, select one or more rows using the checkboxes on the left, then click (Delete). To delete all locations not in use, click Delete All Unused.
Offer Read Access
To show the contents of the table to anyone with read access, place a query web part in the folder or tab of your choice:
Enter > Page Admin Mode.
Select Query from the Select Web Part dropdown in the lower left.
Click Add.
Give the web part the title of your choice.
Select Schema "study" and check the box to show the contents of a specific query.
Select Query "Location" and choose the desired view and choose other options as needed.
A cohort is a group of participants who share particular demographic or study characteristics (e.g., all participants from one country, or all with a particular HIV status). It is common practice to study multiple cohorts of participants to compare these characteristics across larger populations.This topic describes how to configure a LabKey Study to include cohorts, allowing users to filter and display participants by cohort. Studies have a number of built-in facilities an assumptions around cohorts, such as applying them when making 'best-guess' quick visualizations. While you can have multiple participant groupings available in a given study, only one can be used to define cohorts at any given time.
For information on using cohorts once they have been set up, please see the User Guide for Cohorts.
Manage Cohorts
Administrators can access the "Manage Cohorts" page via these routes:
From within a study, click the Manage tab, and then click Manage Cohorts.
From any datagrid, select Participant Groups > Manage Cohorts.
Assignment Mode: Simple or Advanced
Simple: Participants are assigned to a single cohort throughout the study. For example, demographic cohorts set up to track the impact of a given treatment on different categories of individual.
Advanced: Participants may change cohorts mid-study. For example, if your cohorts are based on disease progression or treatment regimen, participants would potentially move from cohort to cohort based on test results during the term of the study. Note that advanced cohort management requires automatic assignment via a study dataset that can include multiple participant/visit pairs (i.e. not a demographic dataset).
Switching between assignment modes requires updating cohort assignments for all participants.
Note that this option is not supported (or shown) for continuous studies, because it relies on the participantVisit or visit tables that are not used by continuous studies.
Assignment Type: Automatic or Manual
Participants can be assigned to cohorts manually, or automatically based on a field you specify in a dataset. If you are automatically assigning cohorts, you will also see a section here for specifying the dataset and field. See Assign Participants to Cohorts for further information.
Defined Cohorts
The Defined Cohorts section lists all cohorts currently defined for the study. You can insert new cohorts, delete unused ones, and export information about current cohorts in various formats.
Note: Use caution not to define any cohorts containing an apostrophe (i.e. a single quote) in their name, whether you are pulling the values from a dataset column or defining them manually. When a cohort contains an apostrophe, internal errors will occur, including but not limited to an inability to use the options on grid menus for any dataset in the study.
Hover over any cohort row to see the (Edit) icon.Edit to specify whether the members of that cohort are considered enrolled in the study, the expected or target subject count, and a short text description of the cohort. When assignment is manual, you can also change the label for the cohort; this option is not available when assignment is automatic based on a data value.If desired, you can also add to the default fields that define a cohort by clicking Edit Cohort Definition and specifying additional fields. The new fields you define can then be specified using the per-cohort Edit links as above.
Participant-Cohort Assignments
The bottom of the "Manage Cohorts" page shows a list of the participants within the current study and the cohort associated with each participant.
Assign Participants to Cohorts
Automatic Cohort Assignment
The Automatic option for mapping participants to cohorts allows you to use the value of a dataset field to determine cohort membership. The field used must be of type string.
Select the name of the mapping dataset ("Demographics" in this example) from the Participant/Cohort Dataset drop-down menu.
Select the Cohort Field Name you wish to use. In this example, there is a column named "Cohort" but another name might be used, such as "Group Assignment".
Click Update Assignments.
To record time-varying cohort assignments, your assignment dataset must be able to contain multiple participant/visit pairs for each participant. Each pair will record the entrance of a participant into a new cohort. It is not necessary to have a data entry for every participant/visit combination. Demographic data, which is collected once per participant in a study, cannot record these time-varying assignments.
Manual Cohort Assignment
If you wish to manually associate participants with cohorts, and do not need to use time-varying cohorts, select the "Manual" assignment type. For each participant, choose a cohort using the dropdown menu which lists available cohorts. Scroll down to click Save when you have finished assigning participants to cohorts.
Use Cohorts
For information on using cohorts and filtering datasets based on cohorts, see the User Guide for Cohorts.
Participant Aliases are used when you need to align disparate data with different IDs representing the same individual. Learn about aliases in this topic: Participant Aliases
When exporting study data, you can obscure the real participant ids by replacing them with randomly generated, alternate ids. You can either specify your own "alternates" or have the system generate random IDs using a prefix you choose. Alternate IDs are still mapped consistently to the same real IDs; there can only be one set of mappings.Alternate IDs are unique and automatically generated for each participant. Once generated, the alternate IDs will not change unless you explicitly request to change them. Multiple publications of the same study will use the same alternate IDs and date offsets.
You can control the prefix and number of digits in the generated alternate IDs: go to > Manage Study > Manage Participants. You can also export a list of the alternate IDs and date offsets from this page.
Click the Manage tab.
Click Manage Alternate Participant IDs and Aliases.
For Prefix, enter a fixed prefix to use for all alternate IDs.
Select the number of digits you want the alternates to use.
Click Change Alternate IDs. Click to confirm; these alternates will not match any previously used alternate IDs.
Scroll down and click Done.
Export/Import Participant Alternates and Offsets
In addition to assigning alternate IDs, you can also shift participant dates to obscure the exact dates but preserve the elapsed time between them. These "Participant Transforms" can be shared among different studies to maintain alignment of these 'obscured' values.
In order to export participant transforms, you must first have set and used both alternate IDs and date shifting in a publish or export from this study.
Once both are set, clicking Export will export a TSV file containing both the alternate IDs and date offset for each participant. If you already have such a file, and want to align a new study to match the same 'shifted' data, click Import to choose and import it from file or via copy/paste.
Change or Merge Participant ID
If you want to change a participant ID within a study, perhaps because naming conventions have changed, or data was accidentally entered with the wrong ID, or was entered for a participant before an alias was added to the alias-mapping dataset, you can do so as follows.
Go to > Manage Study > Manage Participants.
Click Change or Merge ParticipantID.
Enter the participant id to change, and the value to change it to.
Click Preview.
LabKey Server searches all of the data in the folder and presents a list of datasets to be changed. IDs that don't exist in the study yet will show 0. Learn about merging participant IDs in the next section.
Click Merge to complete the participant ID change.
Note that participant alias datasets are never updated by this merge process; they must be maintained manually.
Merging Participant Data
Suppose you discover that data for the same individual in your study has been entered under two participant IDs. This can happen when naming conventions change, or when someone accidentally enters the incorrect participant id. Now LabKey Server thinks there are two participants, when in fact there is only one actual participant. To fix these sort of naming mistakes and merge the data associated with the "two" participants into one, LabKey Server can systematically search for an id and replace it with a new id value.
Decide which participant ID you want to remove from the study. This is the "old" ID you need to change to the one you want to retain.
Enter the "old" value to change (9 digits in the following screencap), followed by the ID to use for the merged participant (6 digits shown here).
Click Preview.
LabKey Server searches all of the data in the folder and presents a list of datasets to be changed.
Click link text in the report to see filtered views of the data to be changed.
If conflicts are found, i.e., when a dataset contains both ids, use the radio buttons in the right hand column to choose which participantID's set of rows to retain.
Click Merge to run the replacement.
Auto-Create New Alias
If your study is date-based and contains a configured alias mapping table, you will have the option to convert the old name to an alias by selecting Create an Alias for the old ParticipantId. When this option is selected, a new row will be added to the alias table. For details on configuring an alias table, see Participant Aliases.
If an alias is defined for an old id, the server won't update it on the merge. A warning is provided in the preview table: "Warning: <old id> has existing aliases".
If a dataset is marked as read-only, a red warning message appears in the status column..
This option is not available for visit-based studies. To merge participant IDs and update the alias table in a visit based study, you must perform these steps separately.
Delete Participant
To remove a participant from a study, either because they have unenrolled or for other reasons, you can select the participant ID here and click Delete.You can also click the link to View participant data to view the participant's data in another tab, letting you confirm first that you are deleting the correct participant.Deleting a participant will:
Delete all rows for that participant from all study datasets
Delete cohort associations for that participant
Remove the participant from any participant groups
Remove rows for this participant from study.participant, study.participantvisit, and study.participantgroupmap
Note that this feature only deletes the participant from the study. There may still be list references, sample type data, or specimens associated with that participant ID. If there are any non-study tables with foreign key constraints referencing any of the study dataset tables, deletion may lead to constraint violations, which will be displayed to the user so they can be addressed.
Related Topics
Publish a Study - Export randomized dates; hold back protected data columns.
Participant Aliases - Align participant data from sources that use different names for the same subject.
Participant Aliases
A single study participant can be known by different names in different research contexts. One lab might refer to the subject using the name "LK002-234001", whereas another lab might use the name "Primate 44". It is often necessary (and even desirable) to let the different providers of data use their own names, rather than try to standardize them at the various sources. LabKey Server can align all the data for a given subject using Participant Aliases.
When data is imported, a provided alias map will "translate" the aliases into the common participant ID for the study. In our example, the two labs might import data for "LK002-234001" and "Primate 44", with the study researcher seeing all these rows under the participant ID "PT-101". It is important to note that there is no requirement that all data is imported using one of the aliases you provide, you can also use the "real" participant ID.LabKey Server's aliasing system works by internally mapping different aliases to a central participant id, while externally preserving the aliases known to the different data sources. This allows for:
Merging records with different IDs for the same study subject
Consolidating search results around a central ID
Retaining data as originally provided by a source lab
Presenting data back to a given audience using their alias
Provide Participant Alias Mapping Dataset
To set up participant aliases, point to a dataset that contains as many aliases as needed for each participant ID. The participantID column contains the 'central' IDs, another column contains the aliases, and a third column identifies the source organizations that use those aliases.
Create a dataset containing the alias and source organization information. See below for an example.
Select > Manage Study > Manage Participants.
Under Participant Aliases, select your Dataset Containing Aliases.
Select the Alias Column.
Select the Source Column, i.e. the source organization using that alias.
Click Save Changes.
Click Done.
Once an alias has been defined for a given participant, an automatic name translation is performed on any imported data that contains that alias. For example, if participant "PT-101" has a defined alias "Primate 44", then any data containing a reference to "Primate 44" will be imported as "PT-101".Once you have an alias dataset in place, you can add more records to it manually (by inserting new rows) or in bulk, either directly from the grid or by returning to the Participant Aliases page and clicking Import Aliases.To clear all alias mappings, but leave the alias dataset itself in place, click Clear All Alias Settings.
Example Alias Map
An example alias mapping file is shown below. Because it is a study dataset, the file must always contain a date (or visit) column for internal alignment. Note that while these date/visit columns are not used for applying aliases, you must have unique rows for each participantID and date. One way to manage this is to have a unique date for each source organization, such as is shown here:
ParticipantId
Aliases
SourceOrganization
Date
PT-101
Primate 44
ABC Labs
10/10/2010
PT-101
LK002-234001
Research Center A
11/11/2011
PT-102
Primate 45
ABC Labs
10/10/2010
PT-103
Primate 46
ABC Labs
10/10/2010
Another way to manage row uniqueness is to use a third dataset key. For example, if there might be many aliases from a given source organization, you could use the aliases column itself as the third key to ensure uniqueness. If an alias value appears twice in the aliases column, you will get an error when you try to import data using that alias.
Import Data
Once the map is in place, you can import data as usual, either in bulk, by single row, or using an external reloading mechanism. Your data can use the 'common' participant IDs themselves (PT-### in this example) or any of the aliases. Any data added under one of the aliases in the map will be associated with the translated 'common' participantID. This means you will only see rows for "PT-###" when importing data for the aliases shown in our example.If any data is imported using a participant ID that is not in the alias map, a new participant with the 'untranslated' alias will be created. Note that if you later add this alias to the map, the participant ID will not be automatically updated to use the 'translated' ID. See below for details about resolving these or other naming conflicts.
Resolve Naming Conflicts
If incoming data contains an ID that is already used for a participant in the study AND is provided as an alias in the mapping table, the system will raise an error so that an administrator can correctly resolve the conflict.For example, if you were using our example map shown above, and imported some data for "Primate 47", you would have the following participant IDs in your study: "PT-101, PT-102, PT-103, Primate 47". If you later added to the mapping "PT-104" with the alias "Primate 47", and then imported data for that participant "Primate 47", you would see the error:
There is a collision, the alias Primate-47 already exists as a Participant. You must remove either the alias or the participant.
Note that clearing the alias settings in a study does not revert the participant ids back to their original, non-aliased values. Alias participant ids are determined at import time and are written into the imported dataset. But you can add a column to your datasets that shows the original ParticiantIds, which can be useful for displaying the dataset to the source organization that knows the participant by the non-aliased id. To display the original ParticipantId values, add the Aliases column to the imported dataset as follows:
Navigate to the dataset where you want to display the original, non-alias ParticipantIds.
Select (Grid Views) > Customize Grid.
Expand the ParticipantID node.
Check the box for Aliases.
Expand the Linked IDs node and you will see a checkbox for each source organization. Check one (or more) to add the alias that specific source used to your grid.
Save the grid view, either as the default grid view, or as a named view.
The dataset now displays both the alias(es) and the original id.
This topic is under construction for the 25.7 (July 2025) release. For the previous documentation of this feature, click here.
In a study, as in other containers, folder permissions are used to control the access of users and groups to the resources in the study, including lists, wikis, reports, etc. In addition, Study Security offers finer-grained control over access to datasets. Administrators can use study security to implement important data access requirements such as cohort blinding, PHI/PII access restrictions, and control over data editing. Access can be granted by security group on a dataset-by-dataset basis.Note that users of the Enterprise Edition can also use field-level PHI annotations to control dataset access on an even finer grained level.Two Study Security types are supported for studies:
Basic: Use folder permissions.
Custom: Add dataset-level permissions to override folder permissions, making it possible for users/groups to have either higher or lower permissions on the datasets than they have on other folder resources as a whole.
Custom, or dataset-level, security is not supported in shared studies.
This topic describes how to configure study security to meet your needs.
Regardless of the security type selected, the following rules apply:
Users with a "Folder Administrator" role can always import, insert, update, and delete rows in every dataset.
All dataset access requires "Reader" permission in the folder, at a minimum.
First, decide what kind of Study Security you require.
Will you allow editing after import? If so, choose the editable branch of the security type. Only users authorized to edit datasets will be able to do so.
Will any users need different access to study datasets than they do for other folder resources? For example, will you have any users who should see lists and reports, but not the datasets themselves? If so, you need Custom security. If not, you can use a Basic option.
Will any users need different access to different datasets, i.e. edit some, read-only for others? If so, you need Custom security. If not, you can use a Basic option.
Type 1: Basic Security with Read-Only Datasets
Uses the security settings of the containing folder for dataset security.
Only administrators can import, insert, update, or delete dataset data.
Users assigned any "Reader" (or higher) role in the folder can read all datasets, but cannot edit, import, update, or delete them.
This is the default security configuration when you create a new study.
Type 2: Basic Security with Editable Datasets
Identical to Basic Security Type 1, except:
Users assigned the "Author" role in the folder can also insert dataset data.
Users assigned the "Editor" role in the folder can also insert, update, and delete dataset data.
Only users with insert and update permissions will see the relevant grid controls.
Type 3: Custom Security with Read-Only Datasets
Allows the configuration of dataset-level access.
Only administrators can import, insert, update, or delete dataset data.
Users with the "Reader" (or higher) role in the folder may also be granted access to read all or some of the datasets. Access is granted by group.
No edit permissions can be granted and insert and edit options are not visible to non-admin users.
Type 4: Custom Security with Editable Datasets
This security type is identical to Type 3, in allowing configuration of read access to some or all of the datasets to users by group. Plus:
Author or Editor permissions on some or all of the datasets may also be granted to security groups.
Configure Study Security
Study security type is set upon study creation, and can be changed and updated at any time. Study administrators should periodically review study security settings, particularly as needs and group memberships change.
In your study folder, click the Manage tab.
Click Manage Security.
On the Study Security page, select the desired Study Security Type.
Click Update Type.
If you are using one of the Basic types, you can skip to the testing section. If you are using one of the Custom types, continue in the next section.
Group Membership and Folder Permissions
Dataset-level security is applied on a group basis; you cannot assign dataset-level permissions to individual users.Before proceeding, check that the groups you need exist and have the access you expect.
Select > Folder > Permissions.
Check for the groups you need on the Project Groups tab.
Switch to the Permissions tab if you also want to assign folder roles to groups. Note that this assignment does not control the group's final access to datasets, it controls their access to non-dataset resources and makes them eligible for dataset permission assignment.
Click Save and Finish.
Configure Dataset-level Security
Select > Manage Study > Manage Security.
When either Custom security type is selected, you will see a Study Security section, with a row for each site and project group.
Specify permissions for each group using the radio buttons.
Edit All: Members of the group may view and edit all rows in all datasets. Only shown with "Custom security with editable datasets" (Type 4).
Read All: Members of the group may view all rows in all datasets.
None: This group is not granted view or edit access to datasets. Group members will only be able to view some summary data for the study, unless they are also members of a different group that is granted higher access.
Note that these options override the general access granted at the folder level.
A group granted "Reader" to the project may be allowed to "Edit All" datasets.
Or, a group granted the "Editor" role in the folder may be restricted to the "Read All" role for datasets.
Individuals who are members of multiple groups will have the highest level of dataset access conferred by the groups to which they belong.
After using the radio buttons to specify security settings for each group, click Update to apply the settings.
If any group assigned dataset access does not already have folder access, you will see a warning. This may not be a problem, as long as the group members are granted read access in the folder in some other way (via a different group or individually).
Below is an example configuration for custom dataset permissions.
Several groups, including "Blinded Group" are given no access to the datasets.
Lab A Group and Lab B Group will have permissions specified per individual dataset.
The Study Group can edit all datasets.
Note the red icon at the end of the Z Group row. As the hover text explains, this group lacks folder-level read permissions to the study itself, so group members will have the assigned access only if they have been granted the Reader role in the folder in some other manner.
Configure Per Dataset Permissions
The Per Dataset Permissions section lets you specify access for specific datasets. This option is available only when there are groups given Per Dataset access. For each dataset you can choose:
None: The group members are not granted access to this dataset.
Reader: The group members are granted read-only access to this dataset.
Author: The group members are granted read and insert access to this dataset.
Editor: The group members are granted read, insert, update, and delete access to this dataset.
Note that other roles may appear in these dropdowns, depending on the modules deployed on your server.
In the column for each group you can grant access for each individual dataset (row) by setting the dropdown to None, Reader, Author, or Editor.
Dataset-Level and Folder-Level Permissions
Folder level permissions are overridden by dataset-level permissions. Regardless of the setting for a group at the folder level, users who are members of that group will have the assigned dataset-level permission on each dataset.For example, using the above image of security settings, if "Lab A Group" has the "Editor" role in the study folder, they would only be able to edit datasets for which they were assigned the "Editor" role in the Per Dataset Permissions. Other datasets are marked as either read or none, meaning they would not have "Editor" access to them. Likewise, if "Lab B Group" has only the "Reader" role in the folder, they would still be able to edit the "LabResults" dataset because of the per-dataset permission granted.Individual users who are members of multiple groups will have the highest level of dataset permissions assigned to any of their groups.
Test Study Security
It is good practice for study administrators to test the security settings after configuring them. To do so, use impersonation of groups and individuals to ensure that users have the expected access.Learn more in this topic: Test Security Settings by Impersonation
Import/Export Security Policy XML
The Study Security policy framework is flexible enough to accommodate complex role and group assignment needs. Once you've created the policy you like, you can export the study security policy as an XML file. That XML file could later be imported elsewhere to repeat the same security configuration. This same file is generated and included when you export a folder archive checking the box for Permissions for Custom Study Security.
Select > Manage Study > Manage Security.
Scroll down to the Import/Export Policy panel.
Click Export to export the current study security settings as an xml file.You can provide an alternate configuration by editing this file or providing your own. To import new settings, use Browse or Choose File to select the desired studyPolicy.xml file, then click Import.
This topic is under construction for the 25.7 (July 2025) release. For the previous documentation of this feature, click here.
Permissions for accessing reports and dataset views default to being the same as the configured permissions on the underlying dataset. However, in some cases you may want to allow users to view aggregated or filtered data in a report or view, without providing access to the underlying dataset. As described in this topic, administrators can configure additional permissions on a specific report or view to grant access to groups who do not have access to the dataset itself.The Report and View Permissions page allows you to explicitly set the permissions required to view an individual Report or View. Users and groups who have at least read access to the folder are eligible for these permissions.
In a study, select > Manage Views.
The Access column shows links for the type of security applied to each view:
public: visible to anyone with read access to the folder
private: visible to the creator only
custom: permissions have been configured per group/user
Click the link for the view of interest:
There are three options for configuring report and view permissions:
Public : This item will be readable by all users who have permission to the source datasets. Notice that while named "public" this does not open access to the public at large, or even to all users of your server. Only users with Reader (or higher) access to this folder are included in this meaning of "public".
Custom : Set permissions per group/user. Use check boxes to grant access to eligible groups.
Private : this view is only visible to you.
You'll see the lists of site and project groups.
Groups with at least "Read" access to the source data through project permissions will be enabled and have an active checkbox.
You cannot grant access from this page to a group that is disabled here and must use the folder permissions page first.
If the selected report has been shared with specific users, you will also see them listed here. Revoke access for a user by unchecking the box. You cannot add individual user access from this page.
Securing Portions of a Dataset (Row and Column Level Security)
How do you restrict access to portions of a dataset, so that users can view some rows or columns, but not others? For example, suppose you want to maintain cohort blinding by prohibiting particular users from viewing the Cohort column in a study; or suppose you want clinicians to view only data for locally enrolled participants, while hiding participants from other localities.
While LabKey Server does not currently support column-level or row-level permissions, you can restrict user access to subsets of a dataset using the following methods:
Create a Filtered Report/View
Disallow direct access to the dataset, and create a report/view that shows only a subset of rows or columns in the dataset. Then grant access to this report/view as appropriate. For details, see Configure Permissions for Reports & Views.
Linked Schemas
In a separate folder, expose a query that only includes a subset of the rows or columns from the source dataset using a Linked Schema. For details see Linked Schemas and Tables.
Protect PHI Columns
When a column contains PHI (Protected Health Information), you can mark it with the appropriate level of protection:
Limited PHI: Show to users with access to a limited amount of PHI
Full PHI: Show to users only if they have access to full PHI
Restricted: Show only to users with access to Restricted information
Premium Features AvailableSubscribers to the Enterprise Edition of LabKey Server can use PHI levels to control display of columns in the user interface. Learn more about this and other Compliance features in this section:
This topic explains how administrators can configure quality control (QC) states for datasets in a study. Managing the flow of incoming data through a quality review process can ensure consistent and reliable results. Providing QC states for data reviewers to use can support any workflow you require.
Define new QC states to be assigned to dataset records.
Control the visibility of records based on QC state.
Once activated, the user interface includes the following:
Data grids display the QC State menu to Readers and Editors.
Data grids can be filtered based on QC state.
Editors can update the QC state of dataset records.
Data imported to the study can be automatically marked with particular QC states depending on the method of data import.
Set Up Dataset QC
In a study folder, select the Manage tab.
Click Manage Dataset QC States.
Currently Defined Dataset QC States. Define any number of QC states and chose the default visibility for these categories by using the Public Data checkbox.
Note: This meaning of "public" data is data visible to users with Read access to the study. It does not literally mean "public" (unless you have granted that access explicitly).
When a QC state is not marked public, data in this state will be hidden if the Dataset visibility option is set to "Public/approved Data".
This list of QC states also includes any Assay QC states defined in the same folder.
Default states for dataset data. These settings allow different default QC states depending on data source. If set, all imported data without an explicit QC state will have the selected state automatically assigned. You can set QC states for the following:
Pipeline-imported datasets
Data linked to this study from assays or sample types
Directly inserted/updated dataset data
Any Datasets that have been incorporated into your study before you set up QC states will have the QC state 'none'.Data visibility. This setting determines whether users see non-public data by default. Readers can always explicitly choose to see data in any QC state.
The default setting is "All data", i.e., all records regardless of QC state will be initially displayed.
Add QC Columns to a Dataset
This section explains how to add QC columns to a dataset, similar to the following:To add QC states to a dataset, pull in columns from the core.QCStates table:
Navigate to the dataset.
Select (Grid views) > Customize Grid.
Under Available Fields, place a checkmark next to the QC State.
Click Save and save the grid as either the default view or as a named view.
Add Color Formatting
To add color formatting to the dashboard, add conditional formatting to the QCStates table:
Go to > Go To Module > Query.
Open the core schema and select QCState.
Click Edit Metadata.
Expand the Label field.
Under Create Conditional Format Criteria, click Add Format (or Edit Formats once some are defined).
User Guide for Dataset QC. Provides an overview of dataset-level quality control, filtering by QC state and updating QC states.
Study Demo Mode
When viewing a study folder, using demo mode hides the participant IDs in many places, including dataset grids, participant views, etc. This is not to be mistaken for true protection of PHI, but can be useful for showing feature demos, making presentations, or taking screenshots for slides that don't unduly expose identifying information.When demo mode is turned on in a study, the participant ID values are displayed as a string of asterisks:Note that these strings are still links, clicking still opens data about that participant, just with the ID obscured.
Turn On/Off Demonstration Mode
To turn on demonstration mode:
Select > Manage Study > Demo Mode > Enter Demo Mode.
Note: Your browser will continue to display participant ID values in the following locations:
the address bar (when viewing individual participant pages or using URL filters)
the status bar (when hovering over links to participant views, etc.)
free-form text that happens to include participant IDs, for example, comments or notes fields, PDFs, wikis, or messages
Remember to hide your browser's address bar and status bar before giving a demo. You should also plan and practice your demo carefully to avoid exposing participant IDs.
First we create an empty container for the study data to live in.
Navigate to a project where you can create a subfolder.
Select > Folder > Management. Click the button Create Subfolder.
Enter a Name for the study, choose the folder type Study, and click Next.
On the Users/Permissions page, click Finish. (You can configure permissions later by selecting > Folder > Permissions.)
Create or Import the Study
You are now on the overview tab of your new study folder. You can either create a study from scratch, or import a pre-existing folder archive containing study objects.
Create a New Study from Scratch
Click Create Study to begin the process of creating a study from scratch. Learn more in:
Click Import Study to import a pre-existing folder archive that contains a study. It may have been previously exported, or created outside the system following specific guidelines. Learn more in:
A dataset contains related data values that are collected or measured as part of a cohort study to track participants over time. For example, laboratory tests run at a series of appointments would yield many rows per participant, but only one for each participant at each time.A dataset's properties include identifiers, keys, and categorizations for the dataset. Its fields represent columns and establish the shape and content of the data "table". For example, a dataset for physical exams would typically include fields like height, weight, respiration rate, blood pressure, etc.The set of fields ensure the upload of consistent data records by defining the acceptable types, and can also include validation and conditional formatting when necessary. There are system fields built in to any dataset, such as creation date, and because datasets are part of studies, they must also include columns that will map to participants and time.This topic covers creating a dataset from within the study UI. You must have at least the folder administrator role to create a new dataset.
The following sections describe each panel within the creation wizard. If you later edit the dataset, you will return to these panels and be able to change most of the values.
Click Advanced Settings to control whether to show the dataset in the overview, manually set the dataset ID, associate this data with cohorts, and use tags as another way to categorize datasets. Learn more in this topic: Dataset Properties.
Continue to define Fields for your dataset before clicking Save.
Basic Properties
Name(Required): The dataset name is required and must be unique.
The name must be unique, must start with a letter or number character, and cannot contain special characters or some reserved substrings listed here.
The dataset name also cannot match the name of any internal tables, including system tables like one created to hold all the study subjects which is given the name of the "Subject Noun Singular" for the study. For example, if your subject noun is "Pig" you cannot have a dataset named "Pig".
Label: By default, the dataset Name is shown to users. You can define a Label to use instead if desired.
Description: An optional short description of the dataset.
Category: Assigning a category to the dataset will group it with other data in that category when viewed in the data browser. By default, new datasets are uncategorized. Learn more about categories in this topic: Manage Categories.
The dropdown menu for this field will list currently defined categories. Select one, OR
Type a new category name to define it from here. Click Create option... that will appear in the dropdown menu to create and select it.
Data Row Uniqueness
Select how unique data rows in your dataset are determined:
Participants only (demographic data):
There is one row per participant.
A value for the participant field is always required.
Participants and timepoints/visits:
There is (at most) one row per participant at each timepoint or visit.
Both participant and date (or visit) values must be provided.
Participants, timepoints, and additional key field:
There may be multiple rows for a participant/time combination, requiring an additional key field to ensure unique rows.
Note that when using an additional key, you will temporarily see an error in the UI until you create the necessary field to select as a key in the next section.
Define Fields
Click the Fields panel to open it. You can define fields for a new dataset in several ways:
The Fields panel opens on the Import or infer fields from file option. You can click within the box to select a file or simply drag it from your desktop.
Supported data file formats include: .csv, .tsv, .txt, .xls, .xlsx.
LabKey will make a best guess effort to infer the names and types for all the columns in the spreadsheet.
Note that if your file includes columns for reserved fields, they will not be shown as inferred. Reserved fields will always be created for you.
Make any adjustments needed.
For instance, if a numeric column happens to contain integer values, but should be of type "Decimal", make the change here.
If any field names include special characters (including spaces) you should adjust the inferral to give the field a more 'basic' name and move the original name to the Label and Import Aliases field properties. For example, if your data includes a field named "CD4+ (cells/mm3)", you would put that string in both Label and Import Aliases but name the field "CD4" for best results.
If you want one of the inferred fields to be ignored, delete it by clicking the .
If any fields should be required, check the box in the Required column.
Before you click Save you have the option to import the data from the spreadsheet you used for inferral to the dataset you are creating. Otherwise, you will create an empty structure and can import data later.
Set Column Mapping
When you infer fields, you will need to confirm that the Column mapping section below the fields is correct.All datasets must map to study subjects (participants) and non-demographic datasets must map to some sense of time (either dates or sequence numbers for visits). During field inferral, the server will make a guess at these mappings. Use the dropdowns to make changes if needed.
Import Data with Inferral
Near the bottom, you can use the selector to control whether to import data or not. By default, it is set to Import Data and you will see the first three rows of the file. Click Save to create and populate the dataset.If you want to create the dataset without importing data, either click the or the selector itself. The file name and preview will disappear and the selector will read Don't Import. Click Save to create the empty dataset.Adding data to a dataset is covered in the topic: Import Data to a Dataset.
Export/Import Field Definitions
In the top bar of the list of fields, you see an Export button. You can click to export field definitions in a JSON format file. This file can be used to create the same field definitions in another list, either as is or with changes made offline.To import a JSON file of field definitions, use the infer from file method, selecting the .fields.json file instead of a data-bearing file. Note that importing or inferring fields will overwrite any existing fields; it is intended only for new dataset creation. After importing a set of fields, check the column mapping as if you had inferred fields from data.Learn more about exporting and importing sets of fields in this topic: Field Editor
Manually Define Fields
Instead of using a data-spreadsheet or JSON field definitions, you can click Manually Define Fields. You will also be able to use the manual field editor to adjust inferred or imported fields.Note that the two required fields are predefined: ParticipantID and Date (or Visit/SequenceNum for visit-based studies). You cannot add these fields when defining a dataset manually; you only add the other fields in the dataset.Click Add Field for each field you need. Use the Data Type dropdown to select the type, and click to expand field details to set properties.If you add a field by mistake, click the to delete it.After adding all your fields, click Save. You will now have an empty dataset and can import data to it.
This topic describes how to import data to an existing dataset in a LabKey study. Authorized users can add data a row at a time, or import data from spreadsheets or other data files. Imported data must match the structure of the dataset into which it is imported.The following import methods are available:
Update and Merge - For any bulk import method, existing rows may be updated.
Insert Single Row
Navigate to the dataset of interest.
Select (Insert Data) > Insert New Row above the dataset grid.
Enter data for the fields.
Fields marked with '*' are required.
Hover over the '?' for any field to see more information, such as the type expected.
If an administrator has limited input to a set of values, you will see a dropdown menu and can type ahead to narrow the list of options.
When finished, click Submit.
Import from File
Navigate to the dataset of interest.
Select (Insert Data) > Import Bulk Data.
To confirm that the data structures will match, click Download Template to obtain an empty spreadsheet containing all of the fields defined. Fill in your data, or compare to your existing spreadsheet and adjust until the spreadsheet format will match the dataset.
By default you will Add rows. Any data for existing rows will cause the import to fail. See update and merge options below.
To import the data from the file, click the Upload file panel to open it.
Click Browse or Choose File to select and upload the .xlsx, .xls, .csv, or .txt file.
To confirm that the data structures will match, click Download Template to obtain an empty spreadsheet containing all of the fields defined. Fill in your data, or compare to your existing spreadsheet and adjust until the spreadsheet format will match the dataset.
If you want to update data for existing rows, select Update rows. To merge both updates and new rows, check the box: Allow new rows during update. Learn more below.
To provide the data either:
Click Upload File to select and upload the .xlsx, .xls, .csv, or .txt file.
Copy and paste your tabular data into the Copy/paste text box. Select the Format (either tab-separated or comma-separated).
For bulk imports, you can select Update rows to update existing data rows during import. By default, any data for new rows will cause the update to fail.Check the box to Allow new rows during update to merge both existing and new rows.If Update rows and Allow new rows during update are both selected:
Data will be updated for existing row identifiers, and new rows will be created for any that do not match.
Row identifiers are determined by how the dataset is keyed, specified under Data Row Uniqueness in the dataset design.
Note that for a dataset with a system-managed additional key, no row updates will occur, as this key is present specifically to ensure all rows are unique when new data is imported.
For rows being updated, each column in the import file (or pasted data) will be checked. If the import contains a value, the existing field in the row will be replaced with the new value.
When datasets are updated, "smart diffing" identifies and imports only the changed rows. In some cases this can result in considerable performance improvement. Only the minimal set of necessary records are updated, and the audit log will record only the rows and fields that have changed. Both values, "before" and "after" the update are recorded in the detailed audit log.
Premium Resource AvailableSubscribers to premium editions of LabKey Server can learn about adding a Bulk Edit button with the example code in this topic:
Note that SequenceNum values cannot be bulk edited, as they are used in aligning subject and visit uniqueness for datasets. Learn more about premium editions
Reload a Study - Replace data in one or more datasets, as well as other study components.
Import From a Dataset Archive
You can simultaneously import several datasets by composing a dataset archive. Using an archive and the LabKey data pipeline gives administrators flexibility about loading datasets from various locations. The archive format also contains a set of properties to control how the import is performed.
To define the location from which the pipeline will process files, follow the instructions in this topic: Set a Pipeline Override. You may use the standard pipeline root or a pipeline override allows you to load files from the location of your choosing.
Create a Pipeline Configuration File
A pipeline configuration file controls the operation of the pipeline job. For dataset archives, the configuration file is named with the .dataset extension and contains a set of property/value pairs.The configuration file specifies how the data should be handled on import. For example, you can indicate whether existing data should be replaced, deleted, or appended-to. You can also specify how to map data files to datasets using file names or a file pattern. The pipeline will then handle importing the data into the appropriate dataset(s).
Note that we automatically alias the names ptid, visit, dfcreate, and dfmodify to participantid, sequencenum, created, and modified respectively.
Dataset Archive File Format
Each line of a dataset archive contains one property-value pair, where the string to the left of the '=' is the property and the string to the right is the value. The first part of the property name is the id of the dataset to import. In our example the dataset id shown is '1'. The dataset id is always an integer.The remainder of the property name is used to configure some aspect of the import operation. Each valid property is described in the following section.The following example shows a simple .dataset file:
1.action=REPLACE 1.deleteAfterImport=FALSE
# map a source tsv column (right side) to a property name or full propertyURI (left) 1.property.ParticipantId=ptid 1.property.SiteId=siteid 1.property.VisitId=visit 1.property.Created=dfcreate
In addition to defining per-dataset properties, you can use the .dataset file to configure default property settings. Use the "default" keyword in the place of the dataset id. For example:
default.property.SiteId=siteid
Also, the "participant" keyword can be used to import a tsv into the participant table using a syntax similar to the dataset syntax. For example:
The properties and their valid values are described below.actionThis property determines what happens to existing data when the new data is imported. The valid values are REPLACE, APPEND, DELETE.
DELETE deletes the existing data without importing any new data. APPEND leaves the existing data and appends the new data. As always, you must be careful to avoid importing duplicate rows (action=MERGE would be helpful, but is not yet supported). REPLACE will first delete all the existing data before importing the new data. REPLACE is the default.enrollment.action=REPLACEdeleteAfterImportThis property specifies that the source .tsv file should be deleted after the data is successfully imported. The valid values are TRUE or FALSE. The default is FALSE.enrollment.deleteAfterImport=TRUEfileThis property specifies the name of the tsv (tab-separated values) file which contains the data for the named dataset. This property does not apply to the default dataset. In this example, the file enrollment.tsv contains the data to be imported into the enrollment dataset.enrollment.file=enrollment.tsvfilePatternThis property applies to the default dataset only. If your dataset files are named consistently, you can use this property to specify how to find the appropriate dataset to match with each file. For instance, assume your data is stored in files with names like plate###.tsv, where ### corresponds to the appropriate DatasetId. In this case you could use the file pattern "plate(\d\d\d).tsv". Files will then be matched against this pattern, so you do not need to configure the source file for each dataset individually. If your files are defined with names like dataset###.tsv, where ### corresponds to the dataset name, you can use the following file pattern "dataset(\w*).tsv".default.filePattern=plate(\d\d\d).tsvpropertyIf the column names in the tsv data file do not match the dataset property names, the property property can be used to map columns in the .tsv file to dataset properties. This mapping works for both user-defined and built-in properties. Assume that the ParticipantId value should be loaded from the column labeled ptid in the data file. The following line specifies this mapping:enrollment.property.ParticipantId=ptidNote that each dataset property may be specified only once on the left side of the equals sign, and each .tsv file column may be specified only once on the right.sitelookupThis property applies to the participant dataset only. Upon importing the particpant dataset, the user typically will not know the LabKey internal code of each site. Therefore, one of the other unique columns from the sites must be used. The sitelookup property indicates which column is being used. For instance, to specify a site by name, use participant.sitelookup=label. The possible columns are label, rowid, ldmslabcode, labwarelabcode, and labuploadcode. Note that internal users may use scharpid as well, though that column name may not be supported indefinitely.
Participant Dataset
The virtual participant dataset is used as a way to import site information associated with a participant. This dataset has three columns in it: ParticipantId, EnrollmentSiteId, and CurrentSiteId. ParticipantId is required, while EnrollmentSiteId and CurrentSiteId are both optional.As described above, you can use the sitelookup property to import a value from one of the other columns in this table. If any of the imported value are ambiguous, the import will fail.
Administrators can view and edit dataset properties either by clicking the name of a dataset from the Manage Datasets page, or by clicking Manage from the grid itself.
Users with the Editor role in the study can edit data but not the properties of the dataset. These users can view the properties page by clicking Details above a dataset grid. They will see the button to Show Import History only.Non-administrators with Reader access (or higher) can click the Details link above a dataset grid to see, but not change, dataset properties.
Action Buttons
Buttons offer the following options to administrators.
View Data: See the current contents of the dataset.
Edit Associated Visits/Timepoints: Select the visits/timepoints where data will be collected.
Delete Dataset: Delete the selected dataset including its definition and properties as well as all data rows and visitmap entries. You will be asked for confirmation as this action cannot be undone. After deleting a dataset, it is advisable to recalculate visits/dates and/or delete unused visits in order to clear any empty visit and participant references that may remain in the study.
Delete All Rows: Deletes all data rows, but the dataset definition and properties will remain. You will be asked for confirmation as this action cannot be undone.
Show Import History(Editor and higher roles): Shows the files that were uploaded to the dataset after its initial creation. If a file was used to create/infer the dataset it will not appear here, nor will records that were added/updated individually.
Edit Definition: Modify dataset properties and add or modify the dataset fields.
Edit Dataset Properties
From the manage page, admins can click Edit Definition to edit the properties:
Name: Required. This name must be unique. It is used when identifying datasets during data upload and cannot match the name of any existing dataset, including system tables such as "Cohort" or the table created for the "Subject Noun Singular" (i.e. the term used for study subjects).
Label: The name of the dataset shown to users. If no Label is provided, the Name is used.
Description: An optional longer description of your dataset.
Category: Assigning a category to a dataset will group it with similar datasets in the navigator and data browser. Select an existing category from the dropdown, or type a new one in this box to add it. Learn more about categories in this topic: Manage Categories.
Data Row Uniqueness
In the Data Row Uniqueness section, select how the dataset is keyed, meaning how unique rows are determined. Note that changing the keying of a populated dataset may not be possible (or desirable).
Participants only (demographic data): There is one row per participant.
For example, enrollment data is collected once per participant.
Each participant has a single date of birth.
Participants and timepoints/visits: There is (at most) one row per participant at each timepoint or visit.
For example, physical exams would only be performed once for each participant at each visit. Fields here would have one measured value for a subject at a time, but there might be many rows for that subject: one for each time the value was measured.
Each participant/visit yields a single value for weight.
Participants, timepoints, and additional key field: If there may be multiple rows for a participant/visit combination, such as assay data from testing a sample taken at an appointment, you would need a third key. Details are below.
Additional Key Field: Select the field that will uniquely identify rows alongside participant and time.
For example, each blood sample taken from a participant at a visit is run through a series of different tests with many results for that person on that day.
Additional Key Fields
Some datasets may have more than one row for each participant/visit (i.e. time) pairing. For example, a sample taken from a participant at that visit might be tested for neutralizing antibodies to several different virus strains. Each test (participant/visit/virus combination) could then become a single unique row of a dataset. See example table below. These data rows are not "legal" in a standard dataset because they have the same participant/visit values. An additional key field is needed.
ParticipantId
Visit
VirusId
Reading
Concentration
PT-101
10
Virus127
3127.877
70%
PT-101
10
Virus228
788.02
80%
You have several options for this additional key:
User Managed Key: If you know that there will always be a unique value in a data column, such as VirusId in the above example, you can use that data column to define row uniqueness.
Select the column to use using the Additional Key Field pulldown, which will list the fields in the dataset.
You will provide the value for this column and manage row uniqueness yourself.
Note that if you also check the box to let the server manage your key, the setting in this dropdown will be ignored and a system key will be used instead.
If a test reports a reading every few minutes, you could use the time portion of the date column.
Depending on how close together your events are, this may or may not provide row uniqueness.
System Managed Key: If your data does not fit either of the above, you can create a new integer or text field that the server will auto-populate to keep rows unique. For example, you might create a new (empty) field called "ThirdKey", and click the checkbox below:
Let server manage fields to make entries unique: When checked, the server will add values to the selected field to ensure row uniqueness. Integer fields will be assigned auto-incrementing integer values; Text fields will be assigned globally unique identifiers (GUIDs).
Note that you should not use this option for existing data columns or if you will be providing values for this field, such as in the above VirusId example.
Note that when creating a new dataset and configuring additional keys, you may see error messages in the wizard as you cannot select a field until you have added it in the Fields section next.You should also use caution whenever editing an existing dataset's keying. It is possible to get your dataset into an invalid configuration. For example, if you have a three-key dataset and edit the keying to be "demographic", you will see unique-constraint violations.
Using Time as an Additional Key
In a date-based or continuous study, an additional third key option is to use the Time (from Date/Time) portion of a DateTime field. In cases where multiple measurements happen on a given day or visit (tracking primate weight for example), the time portion of the date field can be used as an additional key without requiring duplication of this information in an additional column.
Advanced Settings
Miscellaneous options are offered if you click Advanced Settings for dataset properties:
Show dataset in overview: Checked by default; uncheck if you don't want to include this dataset in the overview grid.
DatasetID: By default, dataset IDs are automatically assigned. If you want to specify your own, use this option. All datasets must have a unique dataset ID.
Cohort Association: If this dataset should be associated with a specific cohort only, select it here, otherwise all datasets are associated with any cohort.
Tag: Use tags to provide additional ways to categorize your datasets.
Edit Dataset Fields
Below the dataset properties, you will see the set of fields, or columns, in your dataset. To edit fields and their properties, follow the instructions in this topic: Field Editor.
Add Additional Indices
In addition to the primary key, you can define another field in a dataset as a key or index, i.e. requiring unique values. Use the field editor Advanced Settings for the field and check the box to Require all values to be unique.
Premium Resource AvailableSubscribers to premium editions of LabKey Server can learn more about adding an additional index to a dataset in this topic:
This topic lists the field names that are reserved in a study. The properties listed here are in addition to the reserved dataset system fields, present in every study dataset. The reserved fields each have a special role to play within the study container. Note that there may be additional reserved fields in some circumstances; if you see an error indicating a field name is reserved, try using a variation (such as using "Site_ID" if "SiteID" is reserved). You can use a field label to show the "reserved" name in your grids if desired.
Field
Data Type
Reserved in Visit-based?
Reserved in Date-based?
Description
ParticipantId
Text (String)
Yes
Yes
The default subject identifier, used to hold the unique id for each subject in the study. This default field name can be changed on study creation to a more appropriate value to reflect the nature of the organisms being studied, for example "MouseId".
Participant
Text (String)
Yes
Yes
A reserved field name (but not currently used by the system).
Visit
Integer, Number, or String
Yes
Yes
Provide special display and import behavior in Visit-based studies. When integer or number data is imported under the field name "Visit", the data will automatically be inserted into the dataset's "SequenceNum" field. Matching values will be loaded into any pre-existing visits; non-matching values will result in the creation of new visits. When string data is imported under this field name, LabKey will try to match the string value to an existing visit label. If a match is found, the data will be imported into the matching visit. If no match is found, no data will be imported and an error will be returned.
SequenceNum
Number (Double)
Yes
Yes
A decimal number used to define the relative order of visits in the study.
StartDate (in Demographic datasets)
DateTime
Yes
Yes
When present in Date-based studies, StartDate can be used as a participant-specific start date, overriding the study's generic start date. Relevant calculations, such as time charts and visit events, are calculated using each participant's StartDate, instead of the generic start date. StartDate must be included in a demographic dataset. For details see Timepoints FAQ.
Date
DateTime
No
Yes
Reserved field only in Date-based studies.
Day
Integer
No
Yes
Reserved field only in Date-based studies. In Date-based studies, "Day" is a calculated field, showing the number of days since the start date, either the study's general start date, or the participant-specific start date (if defined).
VisitDate
DateTime
No
Yes
Records the date of the visit. In Date-based studies, when importing data, this field can be used as a synonym for the Date field.
All datasets have pre-defined (and required) base fields. These system fields are used internally to track things like creation and modification, as well as participant and time for integration with other study data.
A user-assigned string that uniquely identifies each participant throughout the Study.
Created
date/time
A date/time value that indicates when a record was first created.
CreatedBy
int
An integer representing the user who created the record.
Modified
date/time
A date/time value that indicates when a record was last modified.
ModifiedBy
int
An integer representing the user who last modified the record.
SequenceNum
float
A number that corresponds to a defined visit within a Study.
Date
date/time
The date associated with the record. Note this field is included in datasets for both timepoint and visit based studies.
VisitDate
date/time
The date that a visit occurred (only defined in visit-based studies).
Hidden Fields
Lists and datasets also have the following hidden fields. To see them, open (Grid Views) > Customize Grid and check the box for Show Hidden Fields.
Hidden Fields in Lists
Name
Datatype
Description
Last Indexed
date/time
When this list was last indexed.
Key
int
The key field.
Entity Id
text
The unique identifier for the list itself.
Folder
object
The container where this list resides.
Hidden Fields in Datasets
Name
Datatype
Description
Lsid
text
The unique identifier for the dataset.
Participant Sequence Num
text
A concatenation of the participant ID and sequence num
Source Lsid
text
A unique identifier for this row.
DSRowId
long int
The row in the current dataset
Sequence Num
other
The sequence number for the visit
Dataset
text
The name of the dataset
Container
text/link
The folder containing the dataset
Visit Row Id
int
The id of the visit
Folder
text
The folder containing the dataset
Note that there may be additional reserved fields in some circumstances. If you see an error indicating a field name is reserved, try using a variation (such as using "Site_ID" if "SiteID" is reserved). You can use a field label to show the "reserved" name in your grids if desired.
Tutorial: Inferring Datasets from Excel and TSV Files
Use the following step-by-step instructions to quickly create and import study datasets from Excel and/or TSV files. This method of creating datasets via study folder import is especially useful for any of the following scenarios:
You want to avoid manually defining all the fields in all your study datasets.
You want to define and import study datasets as an initial, one-time job, and then develop your study using the server user interface.
You want to control your study data using files, instead of adding/updating data using the server user interface.
The overall process goes as follows:
Prepare Files: Assemble and prepare your Excel and/or TSV files to conform to dataset requirements, namely, having (1) required fields (ParticipantId and timepoint fields) and (2) conforming to dataset uniqueness constraints.
Create Study Framework: Create the directory structure and XML metadata files for the study folder archive.
First prepare your Excel and/or TSV files. Each file will become a separate dataset in your study. Each file must meet the following requirements:
Each file must have a column that holds the participant ids, using the same column name across all files. Any column name will work, such as "ParticipantId" or "SubjectId", but whatever name you choose, it must be the same in all your datasets.
Each file must have the same way of indicating the time point information, either using dates or numbers. If you choose dates, then each file must have a column named "Date" (with calendar dates in that column). If you choose numbers, each file must have a column named "Visit" or "VisitId" (with integers or decimal numbers in that column).
File names with trailing numbers (for example, Demographics1.xlsx) may result in unexpected reload behavior.
To control how these files are reloaded, set up a File Watcher reload trigger and use a "name capture group" in the FilePattern property. For details see File Watcher.
If your files include column names with any special characters (including spaces) you should adjust them prior to import so that you can more easily work with your data later. For example, if your data includes a column named "CD4+ (cells/mm3)", you should edit that name to be just "CD4" for best results. After dataset inferral, you can edit the design to include your desired display name in the Label field properties to show the version with special characters to users. Spaces are displayed automatically when you use "camelCasing" in field names (i.e. "camelCasing" would be shown as "Camel Casing").
Prepare your files.
To use this topic as a tutorial, download the following files to use as test datasets suitable for reload:
To make it easier to load your datasets, make an empty study, then export it to the browser, giving you a framework into which to place your own files.
Once you have created the study folder, click Create Study.
On the Create Study page ensure that the Subject Column Name, Timepoint Style, and Start Date are appropriate to the files you are about to import.
For the tutorial files, use the settings shown here:
Click Create Study and you will land on the Manage tab.
Scroll down and click Export Study.
Leave the folder objects and options at their defaults.
Under Export To: select Pipeline root export directory, as individual files.
Click Export.
You will be redirected to the Files web part.
In the Files web part, open the directories export > study > datasets.
The files in the datasets directory are used by the server to know which Excel/TSV files to import. When these files are removed, the server will import whatever Excel/TSV files it finds in the /datasets directory.
In the datasets directory, select and click to delete the following three files.
datasets_manifest.xml
datasets_metadata.xml
XX.dataset (the XX in this filename will be the first word of your folder name).
Add Your Files
Drag and drop your Excel/TSV files into the Files web part, into the export/study/datasets directory (where you just deleted the three files above).
Reload the Study
In the Files web part, click to navigate to the "study" directory and select the file "study.xml".
Click Import Data.
Confirm that Reload Study is selected, and click Import.
On the Import Study from Pipeline page, make no changes, and click Start Import.
The server will examine your files, infer column names and data types, and import the data into new datasets.
Click the Clinical and Assay Data tab to view the datasets. Note that if you change the original Excel/TSV files and reload these changes, the server will replace the dataset data; update is not supported.
Once the datasets are imported, you can define the rest of your study properties, such as cohort information, visualizations, etc.
Troubleshooting
If there is a problem with the reload process, the server will indicate an error in the pipeline job. Click the Error link to see details.
Duplicate Row
Example error message:
10 Apr 2018 09:52:48,534 ERROR: PhysicalExam.xlsx -- Only one row is allowed for each Participant/Visit. Duplicates were found in the imported data.
10 Apr 2018 09:52:48,557 ERROR: PhysicalExam.xlsx -- Duplicate: Participant = P1, VisitSequenceNum = 1.0
This error message means that the PhysicalExam.xlsx file has a duplicate row of data for participant P1 / visit 1.0. Study datasets are allowed only one row of data for each participant/visit combination. If more than one row of data appears for a given participant/visit combination, data import will fail for the entire dataset.The following violates the dataset uniqueness constraint:The following dataset has duplicate subject id / timepoint combinations.
ParticipantId
Visit
SystolicBloodPressure
P-100
1
120
P-100
1
105
P-100
2
110
P-100
3
90
Solution:Edit your Excel/TSV file to either (1) remove the duplicate row of data or (2) define a new visit number to retain the row of data.The following table removes the duplicate row.
ParticipantId
Visit
SystolicBloodPressure
P-100
1
120
P-100
2
110
P-100
3
90
Missing Visit Column
Example error message:
10 Apr 2018 12:54:33,563 ERROR: LabResults.xlsx -- Row 1 Missing value for required property: SequenceNum
This error message arises in a Visit-based study when one of the files does not have a column named 'Visit'. (The server's internal name for the 'Visit' column is 'SequenceNum' -- that's why the error message mentions 'SequenceNum'.)Solution:Ensure that the timepoint information is captured in a column named 'Visit'.
Missing Date Column
Example error message:
10 Apr 2018 13:00:06,547 ERROR: LabResultsBad.xlsx -- Missing required field Date
This error message arises in a date-based study when one of the files does not have a column named 'Date' (for a non-demographics dataset).Solution:Ensure that the timepoint information is captured in a column named 'Date'.
SequenceNum Issues
Example error message:
ERROR: Physical_Exam.xlsx -- Row 1 Missing value for required property: SequenceNum
This error message arises when one of your columns is named "SequenceNum".Solution:Change the column name to Visit or VisitId.
In a study, data is collected over time for subjects or participants. In some cases the exact date of collection is relevant, in others the elapsed time between collections matters more, and in still others, the sequence is more important than either of these.
Data is organized by sequence number provided into a series of defined visits or events. Visit based studies do not need to contain calendar date information, nor do they need to be in temporal order. Even if the data collection is not literally based on a person visiting a clinic a series of times, sequence numbers can be used to map data into sequential visits regardless of how far apart the collection dates are.When setting up a visit-based study, you define a mapping for which datasets will be gathered when. Even if your study is based on a single collection per participant, we recommend that you still define a minimum of two visits for your data. One visit to store demographic data that occurs only once for each participant; the second visit for all experimental or observational data that might include multiple rows per participant visit, such as in the case of results from two rounds of tests on a single collected vial.You have two options for defining visits and mapping them to datasets:
Import Visit Map. Import visit map XML file to quickly define a number of visits and the required datasets for each.
You will continue associating visits with datasets when you upload unmapped datasets or copy assay datasets.Note: Visits do not have to be pre-defined for a study. If you submit a dataset that contains a row with a sequence number that does not refer to any pre-defined visit, a new visit will be created for that sequence number.
Timepoints/Dates
A timepoint is a range of dates, such as weeks or months, rather than a literal "point" in time. The interval can be customized, and the start-date can be either study-wide or per-participant. For example, using timepoints aligned by participant start date might reveal that a given reaction occurred in week 7 of a treatment regimen regardless of the calendar day of enrollment.When you define a study to use timepoints, you can manually create them individually, or you can specify a start date and duration and timepoints will be automatically created when datasets are uploaded.There are two ways timepoints can be used:
Relative to the Study Start Date: All data is tracked relative to the study start date. Timepoints might represent calendar months or years.
Relative to Participant Start Dates: Each participant can have an individual start date, such that all data for given participant are relativized to his/her individual start date. On this configuration, timepoints represent the amount of time each subject has been in the study since their individuals start dates, irrespective of the study's overarching start date. To set up: Include a StartDate column in a dataset marked as demographic.
For example, a study might define 28-day timepoints automatically based on each participant's individual start date using the Demographics dataset. The relevant items in the dataset definition are circled:
Continuous
A continuous study tracks participants and related datasets over time, but without the strong concept of a visit or specific timepoint structure. Participants may enter the study at any point and data is collected in a continuous stream. Primate research centers tracking electronic health records (EHR) for their animals are typical users of continuous studies.Learn more in this topic: Continuous Studies
Change the Timepoint Style for a Study
Once you have selected the timepoint style for your study, you have limited options for changing it within the user interface. If you made a mistake and need to change it right away, and before any data is imported, you can do so.Once a study is underway with the timepoint style "Visit", it cannot be changed.Studies using the "Date" or "Continuous" timepoint styles may be switched between these two types with the following behavior:
When a "Date" study is changed to "Continuous", all existing study.visit and study.participantvisit rows for that container will be deleted.
When a "Continuous" study is changed to "Date", default visits and participantvisit assignments will be calculated and inserted.
When publishing a child study from a study of type "Date" or "Continuous", you will be able to choose either of these types in the publish wizard. The published child will be created with this setting, the original study's type will not be changed.
Visits and timepoints are similar ways of dividing collected data into sequential buckets for study alignment and data analysis. A study will be defined to use one or the other method, and administrators use similar processes for creating and working with either method.A timepoint is a range of one or more days in a study. For instance, you might use weeks, defining timepoints of 7 days.A visit uses Sequence Numbers to order data and can be defined as a range of numbers (single number 'ranges' are allowed). For example, a study may assign a given physical exam the Sequence Number 100, but if the whole exam cannot be completed in one day, follow-up information is to be tagged with Sequence Number 100.1. Data from both of these Sequence Numbers can then be grouped under the same visit if it is defined as a range such as 100-100.9.
Define the properties of the visit or timepoint (see below).
Click Save.
Visit Properties
Label: Descriptive name for this visit. (Optional)
This label will appear in the Study Overview.
Note that if you do not set a Label, the Sequence Number will be shown in grids, including the study overview, but will not explicitly populate the Label column. If the visit is a range, the lowest Sequence Number is used as the label.
VisitId/Sequence Range: Each row of data is assigned to a visit using a Sequence Number.
A visit can be defined as a single Sequence Number or a range. If no second value is provided, the range is the single first value.
When viewing the study schedule, the first value in the range is displayed along with the visit label you define.
Protocol Day(not shown in visit creation UI): The expected day for this visit according to the protocol, used for study alignment.
Description: An optional short text description of the visit which (if provided) will appear as hover text on visit headers in the study navigator and the visit column in datasets.
Cohort(not shown in visit creation UI): If the visit is associated with a particular cohort, you can select it here from the list of cohorts already defined in the study.
Note that when a visit is assigned to a particular cohort, it will only be shown in the study navigator when that cohort (or "All" cohorts) is selected.
Visit Date Dataset and Column Name(not shown in visit creation UI): Where to find the visit date in the data.
Visit Handling (advanced): You may specify that unique Sequence Numbers should be based on visit date. This is for special handling of some log/unscheduled events. Make sure that the Sequence Number range is adequate (e.g #.0000-#.9999). Options:
Normal
Unique Log Events by Date
Show By Default: Check to make this visit visible in the Study Overview.
Associated Datasets: Listing of datasets currently associated with this visit. For each, you can set the dropdown to either Optional or Required.
Timepoint Properties
Label: Text name for this timepoint.
Day Range: Days from start date encompassing this visit, i.e. days 8-15 would represent Week 2.
Protocol Day(not shown in timepoint creation UI): The expected day for this visit according to the protocol, used for study alignment.
Description: An optional short text description of the visit which (if provided) will appear as hover text on visit headers in the study navigator and the timepoint column in datasets.
Cohort(not shown in timepoint creation UI): If the timepoint is associated with a particular cohort, you can select it here from the list of cohorts already defined in the study.
Note that when a timepoint is assigned to a particular cohort, it will only be shown in the study navigator when that cohort (or "All" cohorts) is selected.
Show By Default: Check to make this timepoint visible in the Study Overview.
Associated Datasets(not shown in timepoint creation UI): Listing of datasets currently associated with this timepoint. For each, you can set the dropdown to either Optional or Required.
Hover to View Descriptions
If you add descriptions to visits or timepoints, you will be able to hover over the value in the Visit (or Timepoint) column to view the description.The same visit description is available as a tooltip within the study overview.
Visit and Timepoint Types
A visit or timepoint can be one of the following types:
Screening
Pre-baseline visit
Baseline
Scheduled follow-up
Optional follow-up
Required by time of next visit
Cycle termination visit
Required by time of termination visit
Early termination of current cycle
Abort all cycles
Final visit (terminates all cycles)
Study termination window
Associate Datasets with Visits or Timepoints
To mark which datasets are required for which visits or timepoints, use the Study Schedule view.
From the Manage tab, click Study Schedule.
Click the radio buttons at the intersection of dataset/visit pairs you want to define as requirements.
Click Save Changes.
You can also use the following pathways which may be more convenient in some cases.
Map Datasets to Visits
To specify which datasets should be collected at each visit:
From the Manage tab, click Manage Visits.
Click the edit link for the desired visit.
For each associated dataset listed, you can select Required or Optional from the pulldown.
Click Save.
Map Visits to Datasets
To specify the associated visits for a dataset, follow these steps:
From the Manage tab, click Manage Datasets.
Click the label for the dataset of interest.
Click the Edit Associated Visits button.
Under Associated Visits, specify whether the dataset is required or optional for each visit. By default, datasets are not expected at every visit.
Click Save.
Visit Dates
A single visit may have multiple associated datasets. The visit date is generally included in one or more of these datasets. In order to import and display your study data correctly, it's necessary to specify which dataset, and which property within the dataset, contains the visit date. Alignment of visit dates is also helpful in cross-study alignment.Once one or more datasets are required for a given visit, you can specify the visit date as follows:
From the Manage tab, click Manage Visits.
Edit the visit.
From the Visit Date Dataset pulldown, select the dataset (only datasets marked as required are listed here.
Specify a Visit Date Column Name. The date in this column will be used to assign the visit date.
Click Save.
For a timepoint-based study, if the dataset has a column of type Integer named "Day", "VisitDay" or "Visit_Day", then that value will be stored as the VisitDay.
This topic describes how to edit the properties of visits and timepoints that have been created within your study. Whether they were created manually or inferred automatically when data was imported, you can use the instructions here to customize them.
From the Manage tab in a study, select either Manage Visits or Manage Timepoints.Click (Edit) to the left of the name of the visit or timepoint you want to change.In addition to the properties available when you first create visits or timepoints, you can edit some additional properties and associations.
Visit Properties
A detailed listing of visit properties is available in this topic:
The Protocol Day is the expected day for this visit according to the protocol, used for study alignment. It cannot be set explicitly when defining new visits or timepoints, but can be edited after creation. For a date-based study, the default protocol day is the median of the timepoint range. For a visit-based study, the default is 0.
Cohort
If the visit or time point is associated with a particular cohort, you can select it here. The pulldown lists cohorts already defined in the study. By default, timepoints and visits are not specific to individual cohorts.Note that when a visit or timepoint is assigned to a particular cohort, it will only be shown in the study navigator when that cohort (or "All" cohorts) is selected.
Edit Multiple Visits From One Page
Using the Change Properties link on the "Manage Visits" page, you can change the label, cohort, type, and visibility of multiple visits from a single page. This option is not available for timepoints.Note that this link only allows you to change a subset of visit properties while the "Edit" link lets you change all properties for a single visit at a time.
When manually defining or inferring visits from data does not meet your needs, you can import a visit map in XML format to configure multiple visits in a study in one step.
Paste the contents of the visit map XML file into the box.
Click Import.
If the visit map being imported will result in overlapping visit ranges, the import will fail.
Visit Map XML Format
The visit map lists which visits make up the study, which sequence numbers are assigned to them, and additional properties as required. The same options are available as when you create a single visit in the UI. The format of the imported visit map XML must match the study serialization format used by study import/export.For full details, review the visitMap XML source.
Obtain Visit Map via Export
If you want to use the visit map from one study as the basis for another study, you can export the folder containing the original study. In it, you will find a visit_map.xml file. You can paste the contents of this file into the Import Visit Map box of the new study to create the same visits as in the original.
Example visit_map.xml
The following example defines 4 visits, including a baseline.
When data is collected by different sites and organizations, it can be difficult to keep visit naming standards consistent. The datasets generated might use many different ways of referring to the same visit. For example, the following list of values might all refer to the same visit:
"Month 1"
"M1"
"Day 30"
"First Month"
Instead of editing your datasets to force them to be consistent, you can define visit name aliases that are mapped to sequences numbers/VisitIDs in your study.When you import a dataset containing the alias values, each alias is resolved to the appropriate sequence number via this mapping.Multiple alias names can be mapped to a single sequence number, for example "Month 1", "M1", and "Day 30" can all be mapped to the sequence number "30".
Note: Alias/Sequence Number mapping is only available in visit-based studies, not date-based studies which use timepoints.
Create Alias/Sequence Number Mapping
Prepare your mappings as a tab separated list with two columns, like in the example below. You can map as many names as needed to a given sequence number.
Name
SequenceNum
Name
SequenceNum
Two Week Checkup
14
Three Week Checkup
21
Seven Week Evaluation
49
Seven Week Checkup
49
From the Manage tab, click Manage Visits.
Click Visit Import Mapping
Click Import Custom Mapping
If an existing custom map is already defined you will click Replace Custom Mapping.
Copy and paste your tab separated list into the text area.
Click Submit.
Importing Data Using Alias Visit Names
Once the mapping is in place, you can import data using a name for the visit, instead of the sequence number value. Place the string values/alias in a column named "Visit". On import, the server will convert the string values in the Visit column to integers for internal storage. For example, the following table import physical exam data using string values in the Visit column:
A continuous study tracks participants and related datasets over time, but without the strong concept of a visit or specific timepoint structure available in observational and cohort studies. Participants may enter the study at any point and data is collected in a continuous stream. Primate research centers tracking electronic health records (EHR) for their animals are typical users of continuous studies.
To create a continuous study within the UI, create a new study folder and select Continuous as the "timepoint style".The study still utilizes a start date and may also have an end date, though it is not required. To see and adjust these dates, select > Manage Study.Proceed to create a new study and set up the datasets you require, skipping any steps related to visit or timepoint mapping. Note that the resulting study will not have some features like the study schedule or the ability to have users switch cohorts over time. Creating time chart visualizations based on automatic timepoint tracking will also not always work in continuous studies. However, users defining charts can still choose which dates to use for interval calculations by configuring the measures' end dates as described for time charts.
Change the Timepoint Style for a Study
Once you have selected the timepoint style for your study, you have limited options for changing it within the user interface. If you made a mistake and need to change it right away, and before any data is imported, you can do so.Learn more about making changes once a study is underway in this topic:
Visits and Dates
In date-based studies you can override the generic start date, instead using a participant-specific start date, which will be used to calculate visit positions, time-related reports, and all other alignments in the study, relative to the individualized start dates. To set participant-specific start dates:
Add a column named "StartDate" to a demographic dataset. Do not replace or rename the Date column, as it is still a required column in date-based datasets. Retain the Date column and add StartDate as an additional column.
Example Demographics Dataset with StartDate
ParticipantId
Date
StartDate
Gender
P100
1/1/2017
1/1/2017
M
P200
1/22/2017
1/22/2017
F
P300
2/12/2017
2/12/2017
M
Ensure that the table is marked as a Demographics dataset, instead of a Clinical dataset. (See Visits and Dates.)
Recalculate the timepoints relative to the new participant-specific start dates. (See Manage Visits or Timepoints.)
Why Have Visits Been Created That I Did Not Pre-define?
Visits can be created manually (see Create Visits Manually) or automatically when data is imported. When demographic, clinical, or assay data is imported into the study, the server will create new visits for the data if those visits have not already been pre-defined. For example, suppose that three visits have already been manually defined in a study having the VisitIds "1", "2", and "3". When the following dataset is imported into the study, the server will load three of the records into the pre-existing visits and it will automatically create a fourth visit to hold the last row of data.
ParticipantId
VisitId
Weight
P-100
1
160
P-100
2
165
P-100
3
164
P-100
4
162
How Do I Fix Negative Timepoints?
In a date-based study, negative timepoints indicate that some records in the datasets possess dates that precede the generic start date for the study. To get rid of the negative timepoints, do the following:
Find the earliest date referred to in the datasets. Go to each likely dataset, and examine the Date field using "sort ascending" to find the earliest date in that dataset.
Use this earliest record data in the study as the generic start date for the study. To reset the generic start date, see Study Properties.
Visits and Timepoints can be created in the following ways:
Manually by explicitly defining them before any data is imported to the study.
Automatically by importing data with new dates or visit numbers embedded in the data. When the server parses new incoming dataset records, new timepoints/visits will be created for any new dates/visit numbers it finds in the data.
Can a Visit-based Study be Converted to a Date-based Study? Or Vice Versa?
Generally, the date-based/visit-based choice is a permanent choice. You can switch types at the time of study creation before data is imported, but after data is imported, you can make only limited changes.
"Visit" studies with data cannot be converted to "Date" or "Continuous" studies.
If you find you need to change to or from the "Visit" study type, you could recreate the study in a new study folder, recasting it in a different time-style. Create a new study, choosing the desired timepoint type, and re-import the datasets.
What is the Role of the Generic StartDate Field in Visit-based Studies?
Visit-based studies, which use decimal numbers to indicate visits and time sequences in the study, still allow you to specific a "Start Date" for the study. This date is available purely as "metadata" for the study, but is not used in any of the time-related calculations in the study.
Studies can be exported, imported, and reloaded to make it easy to transfer them between servers, duplicate them within a server, and synchronize them with a primary database. Users can generate and use a folder archive, with additional study objects.A few common usage scenarios:
Studies can be exported and reimported to transfer from a staging environment to a production server.
A snapshot of one study can be exported and imported into several new studies so that they share a common baseline.
A brand new study can be generated with the exported structure (with or without the data) of an existing study. This allows very rapid creation of new studies based on templates.
A study can be exported masking all identifying information enabling the sharing of results without sharing PHI or any patient or clinic details.
Topics
Export a Study - Export to a folder archive, either as exploded files or as a zip file.
Reload a Study - Reload a study manually or via script using the API.
Study Object Files and Formats - This topic explains in detail the formats of the exported archive files, including core data files (TSV or Excel) and XML metadata files.
Folder archives are written using UTF-8 character encoding for text files. Imported archives are parsed as UTF-8.
Related Topics
Publish a Study - Export selected study data to a subfolder, typically for wider distribution. Publishing a study lets you control which, if any, PHI data is included in the subfolder.
Exporting a study is a use case of exporting a folder. Study data and metadata are copied to a set of .xml and .tsv files in a specific folder archive format. You can export in an exploded individual file structure or as a single zipped ".folder.zip" archive.As part of the folder export process, you can choose which study elements are included in the archive, including all the elements available in folder exports, plus datasets, cohort assignments, participant views, etc.
Click the Export Study button at the bottom of the page.
Note: This page is also accessible at > Folder > Management > Export tab, as described in Export / Import a Folder.
Study Folder Objects to Export
When you export a study, you can choose which items to include in the archive. In addition to queries, grid views, reports, and settings available when exporting any type of folder, the study folder enables the export of study related objects and properties.
Select the checkboxes for folder and study objects you wish to export. The list will look something like the following:
Use the Clear All Objects button to clear all the checkboxes (then select a minimal set).
Use the Reset button to reset the default set of checkboxes (and add/remove as needed from there).
Cohort Settings: This option exports the cohort definitions to "cohorts.xml." If defined, SubjectCount and Description for cohorts are included.
Custom Participant View: For a study where the admin has pasted in a custom Participant HTML page, the custom participant view is exported as participant.html.
Datasets: For each category of datasets, the definitions and data can be exported independently. Exporting definitions without data is useful for creating new template studies for new data collection. Exporting data without definitions could be used as part of audit or setting up to refresh data without changing any definitions. Definitions, or the fields and types for the dataset, aka the metadata, will be written to "datasets_manifest.xml". Data exported will be written to .tsv files. See Study Object Files and Formats for more details.
Datasets: Sample Dataset Definitions: Include definitions for linked-sample datasets.
Datasets: Study Dataset Data: Include the data contained in study datasets, including demographic, clinical, and additional key datasets. This category includes all datasets not created by linking assays or samples to the study.
Datasets: Study Dataset Definitions: Include definitions of study datasets (previously known as "CRF Datasets").
Participant Comment Settings: This option exports participant comment settings, if present.
Participant Groups: This option exports the study's participant groups. In addition to label, type, and datasetID, the autoUpdate attribute will record whether the group should be updated automatically. The file generated is "participant_groups.xml."
Permissions for Custom Study Security: Include the custom study security configuration in the archive. Study security policies can also be exported separately; for details see Manage Study Security.
Protocol Documents: This option exports the study protocol documents to a "protocolDocs" folder.
Visit Map: This option exports a "visit_map.xml" file detailing the baseline and visit schedule for the exported study.
Options
If the study folder has any subfolders, you can use the checkbox to Include Subfolders in the export if you like.You can also control the amount of Protected Health Information (PHI) that is included in the export.
Include PHI Columns: Select which PHI columns are included in the archive. Options:
Restricted, Full and Limited PHI: Include all columns.
Full and Limited PHI: Exclude only Restricted columns.
Limited PHI: Exclude Restricted and Full PHI columns.
To exclude all columns marked as PHI at any level, uncheck the checkbox.
Shift Participant Dates: Selecting this option will shift date values associated with a participant by a random participant-specific offset from 1 to 365.
The offset is applied to all dates for a specific participant, maintaining the time between dates but obscuring actual dates.
If you want to exclude certain date fields from this shifting, you can exclude them in the advanced properties for the field. Learn more in this topic.
Export Alternate Participant IDs: Selecting this option will replace each participant ID with an alternate randomly generated ID.
Mask Clinic Names: Selecting this option will change the labels for the clinics in the exported list of locations to a generic label (i.e. Clinic).
Export To...
Select the destination for your export from the following options:
Pipeline root "export" directory, as individual files: In the folder pipeline root, an "export" directory is created if it does not exist. The individual files are exported to this directory.
Pipeline root "export" directory, as zip file: In the folder pipeline root, an "export" directory is created if it does not exist. A zip file archive is exported to that location.
Browser as zip file: Default. A zip file will be downloaded to the destination connected to your browser, such as a "Downloads" folder. You will typically see a box with a link to the export in the bottom of the browser window.
Manage Study Security - Export/import study security policies without a full archive.
Import a Study
This topic is under construction for the 25.7 (July 2025) release. For the previous documentation of this feature, click here.
Importing a study from an archive is the same as importing any folder archive with a few additional considerations. Like other imports, you first create an empty folder of type "Study", then navigate to it before importing. You can import from an exported archive, or from another study folder on the same server.
By default, queries will be validated upon import of a folder archive and any failure to validate will cause the import job to raise an error. To suppress this validation step, uncheck the Validate all queries after import option. If you are using the check-for-reload action in the custom API, there is a 'suppress query validation' parameter that can be used to achieve the same effect as unchecking this box in the check for reload action.
Fail Import for Undefined Visits
When you are importing a folder that includes a study, you can control the behavior if the archive also includes any new visits.By default, new visit rows will be created in the study during import for any dataset rows which reference a new, undefined visit. If you want the import to instead fail if it would create visits that are not already in the destination study or imported visit map, check the box Fail Import for Undefined Visits.This option will cancel the import if any imported data belongs to a visit not already defined in the destination study or the visit map included in the incoming archive.
Overlapping Visit Ranges
If you are importing a new visit map for a visit-based study, the import will fail if the new map causes overlapping visits.
Study Templates
A study minus the actual dataset data can be used as a template for generating new studies of the same configuration, structure and layout. To generate one, create a specific template study with all the required elements but add no data. Exporting an existing study without the data boxes checked and using that to create your template study is one option.When you create a new study, choose the Existing folder option and select your template study.
Study reload is used when you want to refresh study data, and is particularly useful when data is updated in another data source and brought into LabKey Server for analysis. Rather than updating each dataset individually, reloading a study in its entirety will streamline the process.For example, if the database of record is outside LabKey Server, a script could automatically generate TSVs to be reloaded into the study every night. Study reload can simplify the process of managing data and ensure researchers see daily updates without forcing migration of existing storage or data collection frameworks.
Choose the objects to export and under Export to:, select "Pipeline root export directory, as individual files."
The files will be exported unzipped to the correct location from which you can refresh.
Explore the downloaded archive format in the export directory of your file browser. You can find the dataset and other study files in the subfolders. Locate the physical location of these files by determining the pipeline root for the study. Make changes as needed.
Update Datasets
In the unzipped archive directory, you will find the exported datasets as TSV (tab-separated values) text files in the datasets folder. They are named by ID number, so for example, a dataset named "Lab Results" that was originally imported from an Excel file, might be exported as "dataset5007.tsv". If you have a new Excel spreadsheet, "Lab_Results.xlsx", convert the data to a tab-separated format and replace the contents of dataset5007.tsv with the new data. Use caution that column headers and tab separation formats are maintained or the reload will fail.
Add New Datasets
If you are adding new datasets to the study:
First ensure that each Excel and TSV data file includes the right subject and time identifying columns as the rest of your study.
Place the new Excel or TSV files into the /export/study/datasets directory directly alongside any existing exported datasets.
Delete the file "XX.dataset" (the XX in this filename will be the first word in the name of your folder) from the directory.
Manually edit both the "datasets_manifest.xml" and "datasets_metadata.xml" files to add the details for your new dataset(s); otherwise the system will infer information for all datasets as described below.
Reload the study.
Reload from Pipeline
In your study, select > Go To Module > FileContent.
You can also use the Manage tab, click Reload Study, and then click Use Pipeline.
Locate the "folder.xml" file and select it.
Click Import Data and confirm Import Folder is selected.
You can use the manual reload mechanism to populate a new empty study by moving or creating the same folder archive structure in the pipeline root of another study folder. This mechanism avoids the manual process of creating numerous individual datasets.To get started, export an existing study and copy the folders and structures to the new location you want to use. Edit individual files as needed to describe your study. When you reload the study following the same steps as above, it will create the new datasets and other structure from scratch. For a tutorial, use the topic: Tutorial: Inferring Datasets from Excel and TSV Files.
Inferring New Datasets and Lists
Upon reloading, the server will create new datasets for any that don't exist, and infer column names and data types for both datasets and lists, according to the following rules.
Datasets and lists can be provided as Excel files or as TSV files.
The target study must already exist and have the same 'timepoint' style, either Date-based or Visit-based, as the incoming folder archive.
If lists.xml or dataset_metadata.xml are present in the incoming folder archive, the server will use the column definitions therein to add columns if they are not already present.
If lists.xml or dataset_metadata.xml are not present, the server will also infer new column names and column types based on the incoming data files.
The datasets/lists must be located in the archive’s datasets and lists subdirectories. The relative path to these directories is set in the study.xml file. For example, the following study.xml file locates the datasets directory in /myDatasets and the lists directory in /myLists.
When inferring new columns, column names are based on the first row of the file, and the column types are inferred from values in the first 5 rows of data.
LabKey Server decides on the target dataset or list based on the name of the incoming file. For example, if a file named "DatasetA.xls" is present, the server will update an existing dataset “DatasetA”, or create a new dataset called "DatasetA". Using the same naming rules, the server will update (or add) Lists.
root folder.xml study study.xml - A simplified metadata description of the study. myDatasets DatasetA.tsv DatasetB.tsv Demographics.xlsx myLists ListA.tsv ListB.xlsx
Automated Study Reload: File Watchers (Premium Feature)
Premium Resource AvailableSubscribers to premium editions of LabKey Server can enable automated reload either of individual datasets from files or of the entire study from a folder archive.Learn more in this topic:
Note: Study, list, and folder archives are all written using UTF-8 character encoding for text files. Imported archives are parsed as UTF-8. In addition, text exports from grids use UTF-8 character encoding.
Study Section of Folder Archive Directory Structure
Studies have the following directory structure within a folder archive. The structure shown below is provided as an example and excludes most details within folders containing lists, views, etc. The included elements in any archive will differ depending the nature of your study, and which export options are selected.
Note that if you are using the specimens module, you will have additional specimen-related content. Learn more in the documentation archives.
XML Formats
Exporting a study folder produces a zipped folder archive containing a set of XML/XSD files that describe the study's settings and associated data. Some of these files are contained in similarly-named folders rather than at the top level. Key .xml and .xsd files are listed here and linked to their schema documentation pages:
cohorts.xsd -- Describes the cohorts used in the study. A cohort.xml file is exported only when you have manually assigned participants to cohorts.
datasets.xsd -- Describes the study dataset manifest. Includes all study dataset-specific properties beyond those included in tableInfo.xsd. Used to generate dataset_manifests.xml.
study.xsd -- A manifest for the serialized study. It includes study settings, plus the names of the directories and files that comprise the study.
studyDesign.xsd -- Includes studyDesign table information including immunogens, adjuvants, sample types, immunization schedule.
visit_map.xsd -- Describes the study visit map. It is used to generate the visitMap.xml file, which describes the study's visits and includes all of the information that can be set within the "Manage Visit" UI within "Manage Study." For example XML file see Import Visit Map.
tableInfo.xsd -- Describes metadata for any database table in LabKey Server, including lists and datasets. A subset of this schema's elements are used to serialize lists for import/export. Similarly, a subset of this schema's elements are used to generate the datasets_metadata.xml file for dataset import/export. Note that a complementary schema file, datasets.xsd, contains additional, dataset-specific properties and is used to generate and read datasets_manifest.xml during dataset import/export. These properties are not included in tableInfo.xsd because of their specificity to datasets.
When you "publish" a study, you select a subset of its data, typically with the intention of allowing broader access to your research, for example, to colleagues working in a related field, or to the general public. A published study becomes a new study folder on your server; by default a child folder of the source study.
You can select narrowly or broadly from the data in the source study. For example, you might select just a few participants and time points to be included in the published study; or you might select the majority of the data, leaving out just a few distracting elements. Select any subsets from the following aspects of the source study:
Participants
Datasets
Timepoints
Visualizations and Reports
What Happens When You Publish a Study?
Data that is selected for publication is packaged as a new study in a new folder. The security settings for the new folder can be configured independently of the original source folder, allowing you to maintain restricted access to the source study, while opening up access to the new (published) folder. By default, the new folder inherits its security settings from its parent folder.
Protected Health Information
You can provide another layer of security to the published data by randomizing participant ids, dates, and clinic names. You can also hold back specified columns of data based on the level of PHI (Protected Health Information) they represent.For details see Publish a Study: Protected Health Information / PHI.
Publish a Study
To publish a study, follow these instructions:
In the study folder, click the Manage tab.
Click Publish Study.
Note: If previous studies were already published from this one, you will have the option to use Previous Settings as defaults in the wizard.
Use the Publish Study wizard to select desired options, clicking Next after each panel.
General Setup: Enter the general information for the published study:
Name: The name of the new study. The default is "New Study" plus an incrementing integer for subsequent publishes from the same study.
Protocol: Browse to select and upload a protocol document.
Location: By default the published study is created as a child of the source study folder. You can select a different parent location if desired.
Participants: Either select from existing participant groups, or use all participants. Note that you cannot select among cohorts here, only participant groups. Cohorts and cohort assignments will be included in the published study regardless of your selection on this page of the wizard.
Datasets: Choose the datasets to include. Study datasets, as well as those created by linking assay and sample data. Choose whether and how to refresh these datasets:
None: Never refresh the datasets.
Automatic: When the source study data changes, automatically refresh this published version.
Manual: Don't refresh changed source study data until an administrator manually does so.
Visits or Timepoints: Choose the visits or timepoints that you would like to publish. You must select at least one.
Study Objects: Choose any additional study objects you would like to publish:
Assay Schedule
Cohort Settings
Custom Participant View
Participant Comment Settings
Protocol Documents
Treatment Data
Lists: Choose the lists to include. Only lists defined in the local study folder are eligible to be included in the published study.
Queries: Choose the queries to include.
Grid Views: Choose the grid views to include.
Only grid views that have been marked as "Shared with all users" can be published.
Reports and Charts: Choose the reports and charts to include.
Remember to also include the queries/lists/datasets on which the reports you choose depend.
If you have reports dependent on grid views, be sure that the views are shared with all users.
Folder Objects: Choose additional folder objects to publish. Details about these folder objects are available in this topic: Folder Objects
Include PHI Columns: Select the level of PHI to include in the published study.
Click Finish to create the published study.
The published study is created by the data processing pipeline. When the pipeline job is complete, you can navigate to the published study by clicking Complete in the Status column, or by using the project/folder menu.
Republish a Study
When you republish a study, you have the option of starting from the setting used to publish it earlier. For example, an administrator might use exactly the same settings to republish a study with corrected data, or might update some settings such as to publish a 36-month snapshot of a trial in the same way as an 18-month snapshot was published.
Return to the original source study used above. Click the Manage tab.
Click Publish Study.
The first option in the publication wizard is to select either:
Republish starting with the settings used for a previously published study: Click one from the list.
Publish new study from scratch.
Click Next and continue with the wizard. The previous settings are provided as defaults on each panel.
You may want to step through and confirm your intended selections. It is possible that options may have moved from one publish wizard category to another since the original publication.
Study Snapshot
Information about the creation of every published study is stored in the study.studySnapshot table. Only users with administrator permissions will see any data. You can view this table in the schema browser, or add a query web part to any tab.
Return to the home page of your study.
Enter > Page Admin Mode.
Select Query from the Select Web Part dropdown in the lower left and click Add.
Give the web part a title.
Choose the schema "study" and click "Show the contents of a specific query or view".
Select the query: "StudySnapshot" and leave the remaining options at their defaults.
Click Submit.
The default grid includes a column showing all the settings used to publish the study.
You can republish a study using any of the links in the study snapshot.
Publish a Study: Protected Health Information / PHI
When publishing a study, you can randomize or hide specified protected health information (PHI) in the data, to make it more difficult to identify the persons enrolled in the study. You can alter published data in the following ways:
Replace all participant IDs with alternate, randomly generated participant IDs.
Apply random date shifts/offsets. Note that data for an individual participant is uniformly shifted, preserving relative dates in results.
Mask clinic names with a generic "Clinic" label to hide any identifying features in the original clinic name.
Exclude data fields (columns) you have marked as containing PHI.
Selecting this option replaces the participant IDs throughout the published data with alternate, randomly generated ids. The alternate id used for each participant is persisted in the source study and reused for each new published study (and for exported folder archives) when the "Use Alternate Participant IDs" option is selected. Admins can set the prefix and number of digits used in this alternate id if desired. Learn more in this topic:
Selecting this option will shift published dates for associated participants by a random offset between 1 and 365 days. A separate offset is generated for each participant and that offset is used for all dates associated with that participant, unless they are excluded as described below. This obscures the exact dates, protecting potentially identifying details, but maintains the relative differences between them. Note that the date offset used for a given participant is persisted in the source study and reused for each new published study.
To exclude individual date/time fields from being randomly shifted on publication:
Go to the dataset that includes the date field.
Edit the dataset definition.
In the designer, open the Fields section.
Expand the date field of interest.
Click Advanced Settings.
Check the box to Exclude From "Participant Date Shifting" on export/publication.
Click Save.
Include (or Exclude) PHI Fields
Fields/columns in study datasets and lists can be tagged as containing PHI (Protected Health Information). Once fields are tagged, you can use these levels to exclude them from the published study in order to allow data access to users not permitted to see the PHI.Learn more about tagging fields as containing PHI in this topic:
When you publish a study, use the radio buttons to indicate the PHI level to include. To exclude all PHI, select "Not PHI".For example, if the study is published including Limited PHI, then any field tagged as "Full PHI" or "Restricted PHI" will be excluded from the published version of the study.
Mask Clinic Names
When this option is selected, actual clinic names will be replaced with a generic label. This helps prevent revealing neighborhood or other details that might identify individuals. For example, "South Brooklyn Youth Clinic" is masked with the generic value "Clinic".All locations that are marked as a clinic type (including those marked with other types) will be masked in the published data. More precisely, both the Label and Labware Lab Code will be masked. Location types are specified by directly editing the labs.tsv file. For details see Manage Locations.
When you create a published study, a new folder is created to hold the data selected for publication. But what if the data in the original study changes? How do you refresh the data in the published study? This topic outlines the options.
Note that if you are using the specimen module, you will have additional specimen-related options. Learn more in the documentation archives.
Definitions
Note that refreshing data is different from reloading data.Refreshing data applies specifically to published studies with a base study elsewhere on the server.Reloading data refers to the process of updating study data from source files. For details on reloading, see Export/Import/Reload a Study.
Which Studies Can Be Refreshed?
The following table summarizes what sorts of datasets are produced when creating published studies.Snapshot datasets can be refreshed to pickup changes in the original study. The design of snapshot datasets cannot be modified after they are created. If you wish to control the design of a dataset, use study publication and select the refresh style "None".
Creation Mechanism
Refresh Style
Datasets Produced
Publish
None
Non-Snapshot Datasets
Publish
Auto
Snapshot Datasets
Publish
Manual
Snapshot Datasets
Refresh Dataset Snapshots
Refresh the data in a published study by updating individual dataset snapshots. Refreshing all dataset snapshots simultaneously is not supported. Note that changing the population of participant groups in the parent study has no effect on the child studies. That is, adding or removing participants from participant groups is not reflected in child studies.To update the snapshot:
Navigate to the dataset in the child study that you would like to update.
Select (Grid Views) > Edit Snapshot.
Click Update Snapshot.
Updating will replace the current data with a new set based on the current contents of the parent (source) study. Confirm that this is your intent.
You can also change whether this data is automatically refreshed from this page.
Learn more about working with dataset snapshots in this topic: Query Snapshots.
This topic describes one way to blind study managers to cohort assignments using LabKey's study features.
Scenario
You are a researcher who is studying the effects of different treatment regimens on HIV. You have assembled a team of colleagues to help you do the work. The team is divided into different roles: nurses, lab technicians, data assemblers, data analysts, etc. To ensure no bias creeps into the results, you want the study to be "blinded", that is, you want the study subjects, the study managers, and the data assemblers to be unaware of which treatments are being administered to which subjects.The blinding technique described below denies team members access to the dataset which holds the cohort assignments, including any queries and visualizations that make use of the cohort assignments. Note that this technique is one of many possible solutions to the problem. See the last section of this topic for other ways to achieve a blinded study.
(One Way to) Set Up Cohort Blinding
Create a dataset that contains the cohort assignments, for example:
SubjectId
Date
Cohort
S-01
2000-01-01
A
S-02
2000-01-01
A
S-03
2000-01-01
B
You can use any name for this dataset, but for this tutorial, we will call it "CohortAssignments".
Set the Data Row Uniqueness property for the CohortAssignments dataset to "Participants only (demographic data)". The "only" means this is a dataset with only one primary key field: ParticipantId, instead of the default two key fields, ParticipantId & Timepoint. Participant-only datasets are used to capture participant features that are measured once in the study, because they cannot change, or are unlikely to change, such as Date of Birth, eye color, cohort membership, etc.
Create a project-level security group of blinded users. Populate this group with the team members that should be unaware of the cohort/treatment assignments used in the study. This group can have any name, but for this tutorial we will call it "Blinded Group".
The solution above is just one way to achieve blinded studies. Other options include:
Assemble the study data into a folder which contains no cohort or grouping information at all. Clone the study by publishing, and introduce cohort information only into the cloned study.
Use the compliance features. Set the cohort-bearing column as some level of PHI data, and control access to that column as you would with PHI data.
Use Participant Groups instead of Cohorts to group the subjects. Make the Participant Groups unshared groups, available only to you.
Shared datasets and timepoints are experimental and advanced features. Please contact LabKey if you would like to use these features to their fullest.
Shared datasets and timepoints can be enabled at the project level and used in different ways for cross-study alignment. Either or both may be enabled in a top-level container (project) of type Study or Dataspace, meaning that studies within that project will be able to share definitions and combined data views can be created at the project level.
Shared datasets and timepoints can be used in different ways for cross-study alignment. When enabled in a project, subfolders of type "Study" within it will be able to make use of:
Shared Datasets: When this option is enabled, all studies in the project will see the definitions of any datasets defined in the root folder of this project.
This setting applies to the dataset definition, meaning that studies will all share the structure, name, and metadata associated with each dataset defined at the project level. This means you can ensure that the same table definitions are being used across multiple containers, and control those definitions from a central location.
Optional sharing of demographic data in the shared datasets is separately controlled.
Studies in the project may also have their own additional datasets that are not shared.
Any changes you later make to the shared datasets at the project level will "cascade" into the child studies. For example, fields added to the dataset definition in the project will also appear in the child studies.
Shared Timepoints: This option applies to the sharing of either visits or timepoints, depending on how the top level project is configured. All studies will use the same kind of timepoint.
When this option is enabled, all studies in the project will see the visits/timepoints defined in the root folder of the project.
This enables aligning data across studies in the same time-series "buckets".
Any changes you make to visits/timepoints in the parent project, including additions, will cascade into the child folder.
In addition, when Shared Datasets are enabled, administrators can make use of an additional sharing option:
Share demographic data: For any dataset defined to be demographic, meaning the Data Row Uniqueness is set to Participants only, this setting is available to enable data sharing across studies, instead of only sharing of the dataset definition. Options:
By default, each study folder 'owns' its own data rows in this dataset and the data is not shared.
When you choose Share by ParticipantId, data rows are shared across the project and studies will only see data rows for participants that are part of that study.
Data added to the dataset in a child study is automatically added to the shared dataset in the parent project, creating a union table out of the child datasets.
Take note that if you have a combination of shared datasets and 'nonshared' datasets defined only in a child study, it is possible to "shadow" a shared dataset if you give a child-folder dataset the same name. You also cannot change whether data in a dataset will be shared once you've added any data. It's good practice to design the dataset expectations and configurations prior to adding any data.
Configure Project Level Sharing
First, set up the parent container, or project, which will form the source of the shared information with subfolder studies.
Create a project of type Study.
Once the empty study project is created, click Create Study.
On the Create Study page, define your study properties and scroll down to the Shared Study Properties section. Note that this section is only available when creating a new study in a project; the options will not appear when creating a new study in a folder.
Enable Shared Datasets and Shared Timepoints. While it is possible to use only one or the other, they are most commonly used together.
Shared datasets must be defined in the top-level project. You may also have 'child-study-specific' datasets in child folders that will not be shared, but they must be created with dataset IDs that do not conflict with existing parent datasets.When creating a shared dataset in the project, we recommend manually assigning a dataset id, under Advanced Settings > Dataset ID. This will prevent naming collisions in the future, especially if you plan to create folder-specific, non-shared datasets. Note that the auto-generated dataset id's follow the pattern 5001, 5002, 5003, etc. When manually assigning a dataset id, use a pattern (such as 1001, 1002, 1003, etc.) that will not collide with any auto-generated ids that may be created in the future.
Shared Timepoints (or Visits)
When shared timepoints are enabled, the Manage tab in the top level project will include a link to Manage Shared Timepoints (or Visits). Click to use an interface similar to that for single studies to manage the timepoints or visits.The timepoints and visits created here will be shared by all study folders in the project.
Share Demographic Data
Once shared datasets and shared timepoints have been enabled, you can enable sharing of demographic data, not just the dataset definitions.For demographics datasets, this setting is used to enable data sharing across studies. When 'No' is selected (default), each study folder 'owns' its own data rows. If the study has shared visits/timepoints, then 'Share by ParticipantId' means that data rows are shared across the project and studies will only see data rows for participants that are part of that study.Enabling data sharing means that any individual records entered at the folder level will appear at the project level. In effect, the project level dataset become a union of the data in the child datasets. Note that inserting data directly in the project level dataset is disabled.
Navigate to the dataset definition in the top level project.
Edit the dataset definition.
In the dataset designer, ensure Data Row Uniqueness is set to Participants Only (demographic data).
Click Advanced Settings, then use the dropdown Share Demographic Data:
No (the default) means that each child study folder 'owns' its own data rows. There is no data sharing for this dataset.
Share by Participants means that data rows are shared across the project, and studies will only see data rows for participants that are part of that study.
Create Study Subfolders
Create new subfolders of this project, each of type Study. You can create these 'sub studies' before or after starting to populate the shared structures at the project level.Note that the parent container has already set the timepoint type and duration, which must match in all child studies, so you will not see those options when you Create Study.Each of the 'sub-studies' will automatically include any definitions or timepoints you create at the project level. It is best practice to define these shared structures before you begin to add any study data or study-specific datasets.
The Data Sharing Container
Note that the project-level container that shares its datasets and timepoints with children sub-studies does not behave like an "ordinary" study. It is a different container type which does not follow the same rules and constraints that are enforced in regular studies. This is especially true of the uniqueness constraints that are normally associated with demographic datasets. This uniqueness constraint does not apply to datasets in the top-level project, so it is possible to have a demographics table with duplicate ParticipantIds, and similar unexpected behavior.If the same ParticipantId occurs in multiple studies, participant groups may exhibit unexpected behavior. Participant groups do not track containers, they are merely a list of strings (ParticipantIds), and cannot distinguish the same ParticipantId in two different containers.When viewed from the project-level study, participants may have multiple demographics datasets that report different information about the same id, there might be different dates or cohort membership for the same visit, etc.
When you create a study dataset, LabKey creates a corresponding table in the main database to hold the data. This table is created in the studyDataset schema. For example if you create the table
"Demographics"
LabKey will create a corresponding table with a related name (container id string + Demographics), for example:
"c10d79_demographics"
Import Process
When users import data to the Demographics table two things occur:
The data is inserted into the studyDataset schema, namely into the corresponding database table "c10d79_demographics".
The data is inserted into the study schema, namely into an extensive number of bookkeeping and junction tables. These tables are the pre-existing, system tables of the study schema, designed to support the application logic and the built-in reports provided by a LabKey Study. (As such, users should treat these tables as "read-only", and not manipulate them directly using PGAdmn or some other db tool. Users should view, edit, and design datasets only via the official LabKey mechanisms such as the study web UI, domain designer, APIs, archive import, etc.)
System Tables
These system tables are not created by the user, but already exist as part of the study schema, for example:
Cohort - Contains one row per defined cohort.
DataSetColumns - Metadata table containing one row of metadata for each column in all study datasets.
DataSets - Contains one row of metadata per study dataset.
Participant - Contains one row per subject. ParticipantIDs are limited to 32 characters.
ParticipantVisit - Contains one row per subject/visit combination in the study. Note that this table is populated dynamically as dataset, assay, and other aligned data is loaded, and is guaranteed to always be complete.
and so on...
Displaying Visit Labels
The ParticipantVisit table contains one row per participant/visit combination in a study. This can be a useful table for aligning data in queries and reports. Note that when you view "ParticipantVisit/Visit" in the UI, the display for that column will show the visit label if one exists, or the SequenceNum for the field if a visit label has not been defined. This is because the grid view will "follow the lookup" to an appropriate display field. However, if you include the "ParticipantVisit/Visit" column in a query, you will not see the the same 'followed lookup', but will instead see the rowID of that row in the table. To use the actual label in your query, you need to follow the lookup, i.e. use "ParticipantVisit/Visit/Label" instead.
Data Integration
Data imported from different studies is intermingled in these system-level study schema tables. (The data is not intermingled in studyDataset schema tables.) To get a sense of how data is intermingled in these tables, open PGAdmin and drill down to:
Databases > labkey > Schemas > study > Tables.
Right-click a table such as "study" or "dataset" and select View Data > View All Rows.
Notice that data from many apparently different datasets and study containers is intermingled there. (When the server displays data, it generally separates it by container, providing a form of data partitioning.)Below is an example system table named "participant". Notice the "containerid" column which indicates which study folder a given participant belongs to.
Premium Feature — Available in the Enterprise Edition of LabKey Server. Learn more or contact LabKey.
Electronic Health Records track the health and behavior of individuals, often non-human primates, and provide critical data in research laboratory environments. The EHR module streamlines population management by ensuring that required care and treatment is provided to the animals and that the necessary data is collected. By tailoring your interface to the equipment used and UI expected of your researchers, you can ensure the accurate recording of health information.LabKey supports a suite of tools to support the management and integration of EHR data. Typically the implementation of these tools is customized to suit an individual research center.On the Overview tab, users have quick access to main features, key queries, and common searches. A typical dashboard might include both a set of main 'task buttons' and a panel of clickable links to provide quick access to specific queries relevant to the given center.
Premium Resources AvailableSubscribers to premium editions of LabKey Server can learn more in these topics:
LabKey's SND module is not included in standard LabKey distributions. Please contact LabKey to inquire about obtaining this module and support options.
The Structured Narrative Datasets (SND) module allows for hierarchical data structures to be defined dynamically by end users to store virtually any type of research or clinical data including full electronic health records, in a contextually meaningful manner that can be easily quantified, analyzed, and billed.
Using the SND module eliminates the need to create new database structures and the associated programming required to support them. This means that the changing needs of scientists can be addressed quickly by domain experts without the need for additional programming to support how data are entered, validated, stored, or queried for.The hierarchical nature of the design allows large, multi-workflow procedures, such as surgeries, to be defined as single units of work that are made up of smaller units. You create individual workflows that can then be combined and reused to build larger workflows so that data is entered and coded in a consistent manner.
Components
Individual workflows are defined using design templates called packages, representing a piece of the workflow and data collection needs. Packages can be combined into super-packages, which can also contain other super-packages.Projects define the group of packages and/or super-packages available to a given study, used for access control, and linking to billing and protocol information.An event stores the contextual information around the data entry for a participant, including the package hierarchies used to define the procedures that were performed.
In a Structured Narrative Dataset (SND) system, individual workflows are defined using design templates called packages, developed by expert users who are familiar with the workflow and data collection needs. Packages can be combined into super-packages, which can also contain other super-packages.
The same super-package can be identified as more than one of the following types depending on the point of view.
Primitive: A package with neither parent nor child packages. It typically defines a particular type of data to be collected, such as a participant's weight.
Top-level package: A package with a null parent package ID and zero or more sub-packages. A primitive can be used as a simple top-level package.
Sub-package: A package contained within a super-package (has a non-null parent package ID).
Super-package: A package with children. It may or may not also be a sub-package or top-level package.
Packages and Categories can be created, edited, and deleted from the user interface.
Packages Grid
The package landing page contains a grid populated with all existing packages.
To create a new package, click New Package.
Typing into the search box will filter the list of packages by ID or description as you type.
The Options menu contains Edit Categories if you have permission to do so.
Use the Show Drafts checkbox to control whether draft packages are shown. Draft packages are those with "Active" set to false.
Each existing package has three action icons:
(eye): View.
(pencil): Edit.
(copy): Clone. Cloning will copy the chosen package into a new package creation form for modifying and saving.
If a package is not "in use" hovering over the row will reveal a (delete) icon on the right.
Create and Edit Packages
The interface for creating, viewing, and editing packages contains the following fields:
PackageID: populated automatically, starting with 10000
Description: This value is passed through to the Packages table.
Narrative: This field contains a string with tokens defined with curly braces ({token}). Each token included here represents data to be entered and will become a row in the Attributes grid. After entering the Narrative, click Parse Attributes as circled above to create the grid where you can set properties of each token:
Attributes: Properties for the tokens included in the Narrative are populated here. Properties include:
Lookup Key: A list of lookups defined in either snd.LookupSets and snd.Lookups, or a registered table or schema, see SNDManager.registerAttributeLookups.
Data Type: Select from the drop down of possible data types.
Label: Textbox input
Min and Max: Specify an expected range of entries.
Order: Select Move Up or Move Down to reorder the rows.
Required: Checkbox.
Redacted Text: Enter the value to be shown in a narrative when data for this attribute is redacted.
Note the "Allow multiple instances of this package in one super package" checkbox. Check to allow this behavior.A package definition after parsing attributes might look something like this:
Narrative Templates
Each package definition includes a narrative template including curly brace delimited {tokens} marking where data will be inserted. Tokens become attributes, into which data is entered.
Package Narrative Template: Defined with each package to provide a template for putting the data into test format. Tokens mark where to put actual data in the template
In a Structured Narrative Dataset (SND) system, organizing packages into categories can help users identify related packages and superpackages.
Categories
Select Categories
The Categories widget on the package definition page shows the selected categories in a text box. Click the box to see a list of available categories (with the selected options shown at the top). Start typing to filter the list.
Define or Edit Categories
When a user has permission to edit or add categories, they will see an Edit Categories option on the packages grid under the Options menu.Click Add Category to add a new one. Only categories that are not in use by packages can be edited, and changes are made inline. Click Save when finished making changes.
In a Structured Narrative Dataset (SND) system, super-packages are composed of other packages; the child packages are assigned to the super-package. Super-packages may also contain other super-packages.
Create Super-packages
When viewing or creating a super-package, use the lower half of the interface to view current assignments in the lower right Assigned Packages panel. In the Available Packages panel to the left, you can type ahead to filter the list of packages or search for a specific package by name or id number.Select any package to reveal a menu. Choose:
Add to add this the package to the super-package. It will then appear in the Assigned Packages panel as indicated in the above screenshot.
View Full Narrative will show the narrative in a popup.
Select any assigned package to reveal a similar menu.
Remove: Remove this package from this super-package.
Make Required: Create a dependency on this package. If a package dependency is defined in this way, the procedures in the "required" packages must be completed during data entry for the super package.
When a package is required, this option will read Make Optional letting you remove the dependency without removing the package. Data entry for this package will then be optional.
Move Down/Up: Reorder the packages.
View Full Narrative will show the narrative in a popup.
In a Structured Narrative Dataset (SND) system, Projects define which packages are available for use in a particular study.
Projects
A project is a way to control access, link to billing and protocol information, and give context to a chosen group of super-packages. The set of packages available on data entry forms can be controlled by project lists, specific to the revision of a project the user is working on. See Create Projects below for more information.
Revisions
Projects have revisions with distinct start and end dates that may not overlap. Gaps in time between revisions are allowed. Revision numbers are incremented by one and gaps in numbers may not exist.Revisions allow for significant changes to existing projects, such as funding sources (defined through external linkage) while providing history about a project's previous incarnations.Only the most recent revision of a project can be deleted, and not if it is in use.
ReferenceId
Projects have a ReferenceId field that provides coupling to a query source (table) that exists outside the SND model. This field is used to provide access to additional data that are necessary for the particular implementation of this model. For example, funding sources or participant data. The reference source is defined in the module configuration to make the design more robust by avoiding the need to anticipate the ways in which the model will be used.
Create and Edit Projects
The list of projects can be searched, and shown with or without drafts and inactive projects listed. Links for viewing, editing, or revising each project are provided. Create a new one by clicking New Project.
Project Creation and Editing
The interface for creating and editing projects has a panel for start/end date and description, with a dual panel for selecting packages below it. Click a package on the Available Packages list and select Add to add it to the project. You can view the narrative for any package by clicking it.
In a Structured Narrative Dataset (SND) system, events are the primary objects used to tie together data recorded for a participant. To add data to a participant's record, a project is selected, and a super-package is chosen from its list. The user fills in the data for the attributes defined in the XML narrative, and these values are then stored.Events are linked to the project table to provide a reference to a restricted super-package list. They are also linked to an external query source to validate the participant’s membership in the project.
Data Entry Workflow
Select a participant ID.
Select the appropriate project. The project contains an approved list of procedures (packages) that can be performed and ties those packages to billing and protocol information.
Select the package for which to enter data.
Enter the appropriate data on the given page.
Procedure Attributes
The data entry panel lists procedure attributes to be provided, such as treatment amounts and route of treatment.
Procedure Narrative
The procedure narrative records step by step what has happened and connects the description with the package IDs where the entered data has been recorded. Using a narrative template and entered data filtered to a specific participant and event, the narrative can be generated using the API.
Narrative Generation
Each package definition includes a narrative template including curly brace delimited {tokens} marking where data will be inserted.
Package Narrative Template: Defined with each package to provide a template for putting the data into test format. Tokens mark where to put actual data in the template
Super Package Narrative Template: A text representation, in hierarchical format, of all the Package Narrative Templates.
Example:
- Weight and vitals - Heart rate = {HR} bpm, temperature = {temp} F, respiration rate = {RR} bpm - Weight = {weight} kg
The tokens become attribute columns, populated as part of data entry. Using the templates, event narratives can be generated by populating the templates with the entered data for the specific event and participant ID.
Super Package Narrative For Event: Super Package Narrative Template with the saved values for a particular SND Event on a particular participant Id filled in.
Example:
5/5/2017 12:12:29 Admit/Charge ID: 1111 ** Weight and vitals - Heart rate = 131 bpm, temperature = 100.70 F, respiration rate = 35 bpm - Weight = 7.10 kg
Full Event Narrative: All the Super Package Narratives for an Event for a given participant Id.
Example:
5/5/2017 12:12:29 Admit/Charge ID: 1111 ** TB test right eye. Result: negative ** Weight and vitals - Heart rate = 131bpm, temperature = 100.70 F, respiration rate = 35 bpm - Weight = 7.10 kg
Full Narrative: Full Event Narratives over a given period of time.
Example:
5/5/2017 10:12:29 Admit/Charge ID: 1111 ** TB test right eye. Result: negative ** Weight and vitals - Heart rate = 131bpm, temperature = 100.70 F, respiration rate = 35 bpm - Weight = 7.10 kg 5/23/2017 15:05:00 Admit/Charge ID: 1111 ** Cumulative blood draw of 1.000 mL for 200XY
Procedure Notes
The notes section allows the entry of additional notes which will be stored with the event.When data is entered, fields are filtered to the packages and subpackages associated with the event.
In a Structured Narrative Dataset (SND) system, quality control (QC) and controlling access are key parts of managing data. Assigning a specific QC state to data entered can be used to automatically grant or deny user access based on the state of the data. This topic describes how to set up QC states and assign permission implications to them.
The workflow is 1-2-3 in the case of typical data, and 1-2-4 if the reviewer finds reason to reject the data.QC states are populated via the SND admin UI, described below. This only needs to be done at startup.
Permissions and Roles
Each QC state has its own permissions assigned:
A. Read Permission
B. Insert Permission
C. Update Permission
D. Delete Permission
For example, possible permissions are: In Progress Read Permission, In Progress Insert Permission, In Progress Update Permission and In Progress Delete Permission, etc. Each can be assigned independently of other permissions for the same QC state.To assign these permissions, create the necessary groups of users then assign the following user roles to the groups:
Basic Submitter
In Progress: Read, Insert, Update, Delete
Review Requested: Read, Insert, Update, Delete
Rejected: Read, Delete
Data Reviewer
In Progress: Read
Review Requested: Read
Completed: Read, Update
Rejected: Read, Update
Data Admin
In Progress: Read, Insert, Update, Delete
Review Requested: Read, Insert, Update, Delete
Completed: Read, Insert, Update, Delete
Rejected: Read, Insert, Update, Delete
Reader
Completed: Read
If a user is assigned more than one of these roles, such as by belonging to multiple groups assigned different roles, they will have the union of all individual permissions for each role.
Permission Validation
Permissions are verified on incoming requests. For insert, update and delete, the incoming QC state will be verified against the user's permissions. For example: A Data Reviewer can read 'Review Requested' data, then update the event to either 'Completed' or 'Rejected'.The QC state is updated/validated using the saveEvent and getEvent SND APIs. Permissions are checked before any other kind of validation. Permission checks are done against the categories of the top level packages. If the user does not have the required permission for any of them, then the API action will return an error.
Permissions Assignment UI
To access the permissions assignment UI:
Navigate to the project or folder.
Select > SND Admin.
First, under Controls, click Populate QC States to populate the SND states in the core module. This only needs to be done once at startup.
Next, under Links, click SND Security.
On the Security page, all Categories are listed in rows. The site andproject groups defined are listed as columns. Use the pulldown menus to configure the desired roles for each group on each category.Click Save when finished.
A package is the building block of the SND schema. A package is the detailed representation of a node in the hierarchy, containing metadata about the procedure, workflow step, or other type of data the node represents. The following APIs are defined for package searching, creation, editing, viewing and deletion.SND APIs
savePackage: (New) Saves a new package or update an existing package. Once a package is "In use", determined by the "Has Event" calculated column, then the domain and corresponding property descriptors will not be updatable. Used in the Package UI to save or save draft for new, edit, or clone.
getPackages: (New) Finds one or more packages by their package IDs. Returns an array containing package JSON datas. See JSON for a package below. Used in the Package UI for viewing or initial data for editing or cloning packages.
Other updated or useful APIs to be called on snd.Pkgs table
LABKEY.Query.selectRows: Get a list of package Id’s and abbreviated package info. Used for package list UI. Includes Has Event and Has Project columns for grouping and filtering in the UI.
LABKEY.Query.deleteRows: First checks if package is in use. If so, do not allow to delete. If package is not in use, domain and property descriptors are also deleted.
savePackage
Input:
{ pkgId: int - only if updating or cloning, description: string - (required), active: boolean - (required), repeatable: boolean - (required), clone: boolean - (optional) narrative: string - (required), testIdNumberStart: int - (optional) used for testing only, categories: [categoryId (int), ...] - (optional), attributes: [Property Descriptors] - (optional), extraFields: [Property Descriptors] - (optional) subPackages: [{ - (optional) superPkgId: int - (required), sortOrder: int - (required), repeatable: boolean - (required), required: boolean - (required) }, ...] }
Output: No JSON
getPackages
Input:
{ packages:[ packageId: int, …] (required), excludeExtraFields: boolean (optional) - Default false. Used to speed up metadata queries for the UI. excludeLookups: boolean (optional) - Default false. Used to speed up metadata queries for the UI. includeFullSubpackages: boolean (optional) - Default false. Returns the full super package hierarchy. }
Updated or useful APIs to be called on snd.PkgCategories table
LABKEY.Query.saveRows: Saves a list of all categories, preserving categoryId.
LABKEY.Query.selectRows: Return a list of all categories. Used for category editing as well as for populating category drop down menus. Includes an "In Use" calculated column.
LABKEY.Query.deleteRows: If any categories are deleted, the CategoryJunction table is updated. If categories are "In use" (meaning attached to a package) they cannot be deleted.
Projects
SND APIs
saveProject: (New) Used for creating, editing, and revising Projects. Includes draft state, revision number, and project type.
Create Project: isEdit and isRevision set to false. The revision is saved as zero. Must contain a start date, description and reference ID.
Edit Project: isEdit is set to true and isRevision is set to false. Project ID and Revision are not editable and do not change on edit.
Revise Project: isEdit is set to false and isRevision is set to true. Increments the revision number by one. It does not delete the project being revised. Any fields other than project ID can be updated as part of a revision.
getProject: (New) Select all the project information for a given project ID. See JSON for a Project below. Used in the Project UI for viewing or inital data for editing or revising.
Other updated or useful APIs when called on snd.Projects
LABKEY.Query.selectRows: Get a list of abbreviated project/revision data including the hasEvent field (The snd.Events table contains a reference to the project the event was created toward. The Has Event boolean for each project reflects whether such an event exists.).
LABKEY.Query.deleteRows: Will not allow deleting an in use object (object with an event). Only allows deleting the most recent project. Will delete the approved list items in project items.
saveProject
Input:
{ projectId: int - Only for project edits and revisions, revisionNum: int - Only for project edits and revisions, description: string (required), Active: boolean (required), referenceId: int (required), startDate: string (required) (yyyy/MM/dd), endDate: string (optional) (yyyy/MM/dd), endDateRevised: string (optional) - End date for previous revision if doing revision (yyyy/MM/dd) isEdit: boolean (optional) - default false, isRevision: boolean (optional) - default false, extraFields: [Property Descriptors] - (optional), projectItems: [ (optional) Active: boolean (required), superPkgId: int (required) ] }
Output: No JSON
getProject
Input:
{ projectId: int - (required), revisionNum: int (required) }
getEvent: (New) Retrieves event and attribute data and generates the json for the event. Aggregates event data from snd.events, snd.eventnotes, snd.eventdata and attribute data from the exp module. Flags:
getTextNarrative: boolean (Optional) Default is false. Includes the full event narrative in text form in the returned JSON. Includes newline and tab characters.
getRedactedTextNarrative: boolean (Optional) Default is false. Includes full event narrative with redacted values inserted for attribute values where redacted values are defined. Redacted values are defined using the package creation UI.
getHtmlNarrative: boolean (Optional) Default is false. Includes full event narrative in HTML format in the returned JSON. Includes div and span tags with specific classes to define the data defining the narrative and where CSS formatting should occur
getHtmlNarrative: boolean (Optional) Default is false. Includes full event narrative in HTML format with redacted values inserted for attribute values where redacted values are defined.
saveEvent: (New) Used for creating a new event or editing an existing event. Takes the json representation of an event, including attribute data, and populates snd.events and snd.eventnotes.
validateOnly: boolean (Optional) Default is false. When true, all validation triggers are run, but no insert or update is performed. See: SND: Event Triggers
Other updated or useful APIs when called on snd.Events
LABKEY.Query.deleteRows: If an event is deleted from snd.Events, then its related information in snd.EventData, snd.AttributeData (virtual table), snd.EventNotes and snd.EventsCache will also be deleted.
event: { eventId: int - (optional) only if updating an existing event, date: date string - (required) event date (yyyy-MM-ddThh:mm:ss ISO8601), projectIdRev: string - (required) “projectId|projectRev”, subjectId: string - (required) , qcState: string - (required) , note: string - (optional), extraFields: [Property Descriptors] - (optional) depends on customization, eventData: [{ - (optional) event with no data would not have these. eventDataId: int - (optional) for updating, superPkgId: int - (required), sortOrder: int - (optional) extraFields: [Property Descriptors] - (optional) depends on customization, attributes: [{ propertyId: int - (required) either propertyId or propertyName is required, propertyName: string - (required), Value: string - (required), }, …] } …] }
{ eventId: int, date: date string - event date (yyyy-MM-ddThh:mm:ss ISO8601), projectIdRev: string - “projectId|projectRev”, subjectId: string, qcState: string, note: string, exception: Exception - Only if there are Event level exceptions to report, redactedHtmlNarrative: string - Only if requested, htmlNarrative: string - Only if requested, redactedTextNarrative: string - Only if requested, testNarrative: string - Only if requested, extraFields: [Property Descriptors] - ex. USDA Code, eventData: [{ eventDataId: int, exception: Exception - Only if there are EventData level exceptions to report superPkgId: int, sortOrder: int, narrativeTemplate: string, extraFields: [Property Descriptors], attributes: [{ propertyId: int, propertyName: string, Value: string, propertyDescriptor: Property Descriptor (not for input), exception: Exception - Only if there are AttributeData exceptions to report }, …] } …] }
{ projectId: int, description: string, revisionNum: int, active: boolean, startDate: string - date string (yyyy/MM/dd), endDate: string - only if set (yyyy/MM/dd), referenceId: int, hasEvent: boolean, extraFields: [Property Descriptors] projectItems: [{ projectItemId: int, active: boolean, superPkgId: int ...], }
API Testing Framework for Simulating UI
Developers can test data entry user interface by testing the creation of events and attribute data using the API Testing framework. This JavaScript API for entering and retrieving data defined by a super package and associated with an event. You can test the business rules, all the JSON properties, and error cases for these APIs.The tests iterate through a configurable list of tests in a separate file defined in a standard format. The desired results are stored in an external file for comparison.Examples:
The list of tests is placed in a file named tests.js. When a test compares results to an expected value, the results are defined in data.js.To run the tests, navigate to your SND project and go to "snd-test.view#" there. Click Run Tests.
The Structured Narrative Dataset (SND) module was built as an open data module to gather, organize and store data of any type. To allow customization of the business logic in data gathering, the module has a trigger mechanism that allows customized code in other LabKey modules to validate, update or perform other custom actions when data is entered via the SND saveEvent API.
A physical exam might differ depending on the age of the subject. On saving this data via the saveEvent API, a trigger can inspect the incoming data or query the database for the age of the subject and verify the correct physical exam data is being entered. The trigger can then accept or reject the data and provide an error message back to the API caller.
Blood draws might need to follow specific guidelines based on the health of the subject and history of blood draws. A trigger script could query the database for health information and perform the necessary calculations for the blood draw data being entered and provide a warning if approaching or exceeding any limits.
Data taken with different instrumentation could have different units of measurement. A trigger could inspect incoming data to see the units of measurement and convert the values to a different unit of measurement before saving in the database.
To provide additional auditing beyond the normal SND audit logs, a trigger could insert or log additional data about the event data being entered.
Event Trigger Mechanism
The event triggers are called in the SND saveEvent API, right after verifying category permissions (see SND security) and before any base SND validation. This ensures no information is passed to unauthorized users in warning or error messages, but allows the triggers to perform business logic or validation before the base SND validation and before the data is persisted in the database.The triggers are mapped to package categories. When event data is entered, the packages associated with the data are inspected for categories that map to triggers. Only those triggers for the incoming package categories are executed. This allows package designers to add or remove business logic and validation just by adding or removing categories in the package designer UI.
Order of Execution
Because event triggers can be defined at any level of a super package and multiple triggers can be defined for a single package, the following rules apply to the order of execution.
Execution starts at the deepest leaf nodes and then works up the tree of event datas (super package hierarchy), performing a reverse breadth first search. This allows business logic at the most basic package level to be performed first before the higher super package levels.
Sibling event datas (packages) will execute triggers in the order of their sort order in the super package table. This is the order they are listed in the Assigned Subpackage part of the package UI.
Top level packages will execute triggers in order of superPkgId.
A developer can manually define the order of execution of triggers within a package. For example, if a package contains multiple categories that align with triggers, the user can implement a numerical order of those in the triggers themselves (getOrder function). See Creating Triggers below.
Trigger Factory
Create and Register Trigger Factory (First time setup)
The first step in setting up the SND triggers is creating a trigger factory. This factory class will map category names to trigger classes. This should be created in the module that will be using the SND module (created outside the SND module) and should implement the EventTriggerFactory interface from the SND module. The only function that is required in this factory is the createTrigger function which takes a String that is a category name and returns an Object that implements interface EventTrigger (more on this below). The mapping of categories to event triggers is completely independent of the SND module, and can be customized to fit each usage. A simple way is just to use a case statement mapping the strings to new instances of the triggers.The lifecycle of each SND trigger is intended to be for one insert/update occurrence, so the SND module will call the factory createTrigger function for each package category in the incoming event data for every insert/update occurrence. Because of this, and to prevent issues with artifact data, it is recommended to return a new instance of the EventTrigger classes from the createTrigger function. See SNDTestEventTriggerFactory for an example.Once the factory has been created, it will need to be registered in the SND module. SNDService has the function registerEventTriggerFactory which takes the module containing the trigger factory and a new instance of the trigger factory itself as parameters. Multiple factories can be registered, but only one per module. The trigger factories will be executed in dependency order starting at the leaf nodes.
Creating Triggers
Once the factory has been set up, triggers can be created and added to the factory. The trigger classes should be created in the same module as the trigger factory and implement the EventTrigger interface. Two functions must be implemented from the EventTrigger interface and a third optional function is available,
onInsert(Required): This function is called on fired triggers when new event data is being entered.
onUpdate(Required): This function is called on fired triggers when updating existing event data.
getOrder(Optional): Used when multiple triggers are defined on the same package (multiple categories mapped to triggers assigned to same package). Triggers with order defined will take precedent over those with no order. If getOrder returns null, triggers execute in alpha numeric order of the trigger class. Default is null.
The parameters for these two functions are:
Container: The project or folder being used for the data entry. Use this for container scoped queries or any other container scoping.
User: The user performing the data entry. Use this for permission checks.
TriggerAction: This contains the data that might be of interest for validation and business logic. Those fields are:
Event: Event class in the SND API which is a java representation of the entire event.
EventData: EventData class in the SND API. This is the event data associated specifically with the package which contains the category that this trigger is associated with. While this data is part of the Event, this is provided for convenience and performance to not have to search the Event for the applicable EventData.
TopLevelSuperPackages: This is a map of SuperPackages, from the SND API, that represents the top level super packages in the event. The super packages contain their full hierarchies. The key in the map is the EventData id associated with each package.
SuperPackage: SuperPackage containing the category that mapped to the trigger. This is another convenience and performance field so searching the TopLevelSuperPackages is not required.
ExtraContext: This is a map that can be used to pass data between triggers of the same event insert/update occurrence.
Once the triggers are created, their mapping to a category should be added to the trigger factory in the respective module. Now the triggers are associated with the category and will fire for any packages using that category.
Trigger Validation
Custom validation can be done in event triggers, and JSON passed back to the saveEvent API caller will indicate how many validation issues there were as well as exactly where in the data the validation failed. Assuming the JSON passed into the saveEvent API is parseable, syntactically correct, has required data values and correct types, the saveEvent API will always return the event JSON data back to the caller whether the saveEvent call succeeds or fails. On failure, the JSON data will have an exception at the event level either explaining the reason for failure or giving a count of exceptions within the JSON. These exceptions within the JSON will be exception fields within the data object that caused the exception. For example:
If there is not an exception specifically set at the Event level but there are EventData or AttributeData exceptions, the Event level exception will be set to the highest severity of the EventData or AttributeData level exceptions and the exception message with be the counts of the types of exceptions (as shown above).In the trigger, if validation is deemed to have failed or any kind of message needs to be returned to the caller of the SND saveEvent API, then Event.setException, EventData.setException or AttributeData.setException can be used to set an exception at those levels in the JSON (see JSON example above). The exceptions take a ValidationException which can be three different types:
Error: This is the most severe. If an error is found on any of the data types then the saveEvent will not proceed beyond the trigger validation and will return the input event with the errors.
Warning: This will not cause the saveEvent API call to fail but will return a warning exception with the returned event.
Info: This is the least severe. This will not cause the saveEvent API call to fail but will return an info exception with the returned event.
See example triggers in the SND module, sndsrcorglabkeysndtriggertesttriggers
For developing client-side code for the SND module, using ReactJS, Redux and TypeScript, it is recommended to set up the npm development environment. This allows the UI source files to be automatically rebuilt by Webpack when changes are detected and the updated ReactJS components to be hot swapped into the running web application. This makes it possible to see your changes applied without having to rebuild and refresh, preserving the context of your application.Steps to set up:
Install Node.js and npm. You will want this to match the versions in gradle.properties at the top level of your LabKey project. See the npmVersion and nodeVersion fields in the property file. These are the versions the deployed application will be built and packaged with so it is important to use those versions locally while developing.
If the install doesn't add npm to your PATH, add it. Verify npm is installed and is the correct version by running 'npm -v' in your command line.
Start your development machine in IntelliJ in debug mode.
In the command line on your dev machine, go to the source directory of the SND module.
/server/modules/snd
Execute the command, npm start. This will run the webpack build then start monitoring for any changes in the source code.
In your web browser navigate to the development web address for the SND module,
<labkey context>/<folder>/snd-appDev.view#/
Technology Stack
npm: Package manager for JavaScript. Allows for easier installation and maintenance of JavaScript libraries, frameworks and tools like React.js. Uses package.json to manage dependencies and define commands. Chosen for its broad industry usage and large package registry (477,000).Webpack: Module bundler and processor. Organizes, bundles and processes client-side code. Generally making fewer and smaller bundles to be loaded from the server. Creates a dependency graph to control the order of loading dependencies. Loaders allow different types of files to be bundled (.js, .html, .css, .jsx, .tsx). Good for single page apps and works well with ReactJS and TypeScript.Babel: Compiler for next generation JavaScript. Converts next generation JavaScript to the target version of choice. Allows for usage of more modern syntax and paradigms, while maintaining backward compatibility. For our case, we're transpiling es6 features like arrow functions, JS classes, this scoping and promises back to es5.TypeScript: Superset of JS that allows static typing. Supports interfaces and strong type checking similar to Java, but allows dynamic types like normal JS as well. Helps produce more readable and maintainable code. Transpiles to ES6 and JSX.ReactJS: Create responsive, stateful UIs. Modular components that will update only their own html when their state changes without having to reload the page.Redux: Global state storage. A data store we use for things like storing rows in a grid which we can then filter, sort, etc. A single store, but has tree of objects that must be updated through defined actions. Works well with ReactJS.
Query Backed Snapshots (Experimental Feature)
This feature is experimental and needs to be enabled:
Go to > Site > Admin Console > Experimental Features
Check the box for "Allow query based dataset snapshots"
Source Query
The Snapshot feature is only available within the study schema, so the source query for the snapshot must be created in this schema.
Go to > Developer Links > Schema Browser
In the study schema, choose any table or query then click Create New Query at the top of the schema browser.
Enter a name for the source query. This should not be the name of the dataset you're going to create but maybe something related to that name, ex. weightSource.
Next, write, test and save the source query that will be the basis for the query backed snapshot.
This source query is an opportunity to customize the data being supplied for the dataset.
You can use the recently created SND.Categories schema or any other schema for the query.
Required Columns
In order to fit into the study schema as a dataset the query does have some required columns.
_key: this is the pk of the dataset. If using the SND.Categories or SND.Packages schema this will likely be the lsid column, but this must be specifically defined in the query. See example below.
participantid: Id of the animal. May need to define this in the query from SubjectId.
date: All datasets require a date column.
qcstate: SND qcstate can be used here. Can be pulled directly from SND.Categories or SND.Packages tables.
lsid: This can be pulled directly from SND.Categories or SND.Packages tables.
Example:
SELECT LSID as _key, SubjectId as participantid, * FROM SND.Categories.Weight
Create Snapshot
The next step is to create the snapshot which creates the query backed dataset that will fit into the study and EHR.
After you've saved your source query, navigate to the results grid of the query.
Select > Create Query Snapshot.
Name the snapshot, the name of the dataset that will be integrated into the study and EHR. Ex. weight or vitals
Check Query Backed Dataset (only available if experimental feature is turned on)
Click Create Snapshot
If there are missing required columns, you will be informed at this point. You will need to update the source query and try to make the query based snapshot again following the steps in this section.
Testing and Performance
At this point the dataset is created. This is essentially a mock dataset that just passes through the results of the source query. If the source query is not performant, then the dataset will have the same issues. Queries or reports of varying use cases and complexity could result in different performance, so it's important to test across the usages of that dataset and if necessary optimize the source query.
Each Structured Narrative Dataset (SND) table is created as an extensible table, meaning additional columns can be added to customize the table for particular use cases. For example, for a Primate Research Center a USDA code is needed with each package. This column is not in the base SND table but can be added as a virtual column for the PRC’s implementation of the SND module.
Setup
Adding additional columns is done by defining the columns in XML format, then calling an action to upload and create those columns. To do the initial setup, keep reading, otherwise you can skip to step 5 to add or adjust the columns.There is not currently a general-purpose UI trigger to call that action and identify which XML file to upload, so each implementation will have to add a button or some kind of trigger to initiate the action. Here are the steps to set up the action and XML file.1. Declare a module dependency on the SND module in <module_root>/module.properties.2. Create a <module_root>/resources/domain-templates directory in your module.3. Create an XML file in the domain-templates directory to define the extensible columns. The file must have the name format <file name>.template.xml, where name can be any valid file name and should be unique for the .template.xml files in that module.4. The action that will need to be called to upload and create the additional columns is LABKEY.Domain.create. In the module using the SND module, create a button or some other event trigger and set the action to call LABKEY.Domain.create. The JavaScript function should look something like this.
createDomain: function () { LABKEY.Domain.create({ module: <name of your module>, domainGroup: <file name>, // This is the name defined in step three. domainKind: "SND", importData: false, // Not importing data into new columns success: this.successFn, // Function called after columns successfully uploaded. failure: this.failureFn, // Function called if there is an error in the upload. // Exception passed as parameter }); }
5. Add your column definitions to the template XML file. The XML for this file is defined in domainTemplate.xsd and tableInfo.xsd. See test.template.xml in the SND module and snd.template.xml in the SNPRC module for examples. Template documentation can be found here.6. When your column definitions are ready, click the button to create the domains. The entire XML file (may include multiple tables) will be uploaded and the virtual columns will be added, removed or edited based on the prior state of the tables.
Technical Details
Each SND table has a DomainDescriptor, which provides table level metadata about the SND table and allows it to be linked to a set of PropertyDescriptors.
The PropertyDescriptor table contains the column level metadata which can define new columns or supplement existing columns. While the actual data stored in these columns can be stored in any table, this implementation stores the data in an existing table, ObjectProperty. This table is a generic data storage table in the exp schema which essentially just stores the name, value and data type for each row of data.
When populating the LabKey Server SND system, XML import can be used to define attributes, packages, and superpackages via packages.xsd (along with columntype information drawn from tableInfo.xsd). The following tables can have data imported via this mechanism:
snd.packages
snd.superpackages
exp.domaindescriptors
exp.propertydomain
exp.propertydescriptors
To import, select the XML file in the pipeline, click Import Data and select the Packages import option.
Note: package information can be added and updated via XML, but it cannot be deleted. This enables incremental updates using this facility.
Specimen Tracking (Legacy)
Premium Resource - Specimen functionality is a legacy feature available only to existing users of it with a Premium Edition of LabKey Server.
Specimen functionality is no longer part of the study module. The standalone specimen module must be obtained and installed to use specimens with studies. This module is only available to current Premium Edition subscribers who are already using it.You can find documentation of the specimen module and accompanying legacy features in the documentation archives here:
Panorama, implemented as a module for LabKey Server, provides web-based tools for targeted mass spectrometry experiments that integrates into a Skyline SRM/MRM/PRM/DIA workflow. Skyline and Panorama can be used for complete monitoring of your LC-MS system performance.Panorama offers the following solutions for targeted mass spec research:
Easy aggregation, curation, and secure sharing of results
Guidance for new experiments based on insights from previous experiments
Search and review over a large collection of experiments, including multiple versions of documents
Automated system suitability checks, tracking a variety of metrics and reports
Support for both peptide and small molecule analysis
Researchers have two options for using Panorama:
PanoramaWeb provides a public Panorama server hosted at the University of Washington where laboratories and organizations can own free projects. Sign up on PanoramaWeb.
Panorama can be installed by laboratories and organizations on their own servers. It is available as a Panorama-specific Premium Edition of LabKey Server. Learn more or contact LabKey.
Once you have a project on a Panorama server, the topics in this section will help you get started using it for specific uses.
Panorama is a web server database application for targeted proteomics assays that integrates into a Skyline proteomics workflow. The LabKey Panorama module supports management of targeted mass spectrometry data and integration with Skyline workflows (SRM-MS, filtering or MS2 based projects).
To begin working with Panorama, create a new folder, choosing Panorama as the folder type. On the Configure Panorama Folder page, select one of the available configurations. Each type in this list links to a set of documentation.
Experimental data: A collection of published Skyline documents for various experimental designs.
Quality control: System suitability monitoring of reagents and instruments.
Chromatogram library: Curated precursor and product ion expression data for use in designing and validating future experiments.
Check Rank peptides within proteins by peak area if your data contains relative peptide expression for proteins.
Multi-attribute method (MAM): An experimental data folder variant with additional reporting for multi-attribute method analyses.
Create New Panorama Folder
Create a new folder. Hover over your project name in the menu bar (On PanoramaWeb, right below the Panorama icon) and click on the New Subfolder icon circled in the image below.
Give the new folder a Name, then select the Panorama option under Folder Type. This is the folder type that should be selected for all workflows supported by Skyline (SRM-MS, MS1 filtering or MS2 based projects).
On the Users / Permissions page, select one of the available options and click Next.
You can also change permissions on a folder after it has been created.
The next page, Configure Panorama Folder, asks you to choose the type of Panorama folder you would like to create, detailed above.
To import targeted mass spectrometry data, first create the appropriate type of Panorama folder for the work you will be doing. This topic describes how to import data to any Panorama folder.
Open the document that you want to publish to Panorama.
Either click the Upload to Panorama button (circled below) or select File > Upload to Panorama.
Import Data from Panorama Interface
Navigate to your Panorama folder.
Click the Data Pipeline tab.
Click Process and Import Data.
Drag the files or directories into the web part. The upload will begin.
If files are being zipped during upload, the upload progress window will inform the user that it is zipping files.
Zip Files on Upload
Certain file formats and directory layouts used by different instrument vendors can be zipped upon upload to reduce overhead. See the list of recognized patterns below. When you upload a directory or collection of directories that match known mass spectrometry patterns, the files will be zipped. Files or directories that do not match the known patterns will be uploaded as usual.In either case, the progress message will inform the user of the action and status. After the successful upload, the files and/or archives will be available in the file browser.
Browser support: Chrome provides the best performance for zipping upon upload. Edge is not supported.
Recognized File Patterns
The following patterns used by these instrument vendors are supported:
Waters: A directory with the extension '.raw' containing at least one file matching the pattern '_FUNC*.DAT', e.g. _FUNC001.DAT
Agilent: A directory with the extension '.d' containing a subdirectory named 'AcqData'
Bruker: A directory with the extension '.d' containing at least one file of name 'analysis.baf'
Bruker: A directory with the extension '.d' containing at least one file of name 'analysis.rdf'
A Panorama folder of type Experimental data provides a repository of Skyline documents, useful for collaborating, sharing and searching across multiple experiments.
Tutorial
The Sharing Skyline Documents Tutorial on the PanoramaWeb site provides an introduction to using this folder type and covers the following areas:
Requesting a project on panoramaweb.org
Data organization and folder management in Panorama
Publishing Skyline documents to Panorama
Data display options and searching results uploaded to Panorama
Providing access to collaborators and other groups
As proteomics methods are developed and refined, multiple documents are often produced that need to be tracked and linked together. For example, a first runs attempt may include many proteins, precursors, and transitions, which later run attempts will progressively narrow down to the best performing ones. In order to track the method development, documents can be marked with comments and linked together as a series of different versions.
You can view a detailed profile for each document in Panorama by clicking the document name in Targeted MS Runs web part.The Document Summary panel shows key summary information and links to actions and views.
Click the # versions link to jump to the Document Versions panel which shows the document's position in the series of versions.
Download the Document
In the Document Summary, the icon gives the full document size. Click to open a download menu.Options:
SkyP file: A .skyp is a small text file that makes it easy to open the full document in Skyline. Skyline will automatically download the full file from Panorama (if needed). Unlike the .sky.zip file extension for the full Skyline file, Windows associates the .skyp file extension with Skyline so you can just double-click the file.
Full Skyline file
Copy relative URL to clipboard
Copy full URL to clipboard
Automatically Link Skyline Documents
The server will automatically link Skyline documents together at import time, provided that the Skyline documents provide a document ID. When importing a Skyline document whose ID matches one already in the folder, the incoming document will automatically be linked to the previous document(s) as the newest version in the document chain.The document’s import log file will indicate if it was attached as a new version in the document chain.
Manually Link Document Versions
To manually chain together a series of document versions, select them in the Targeted MS Runs web part and click Link Versions.The Link Versions panel will appear. You can drag and drop the documents into the preferred order and click Save.Note that an individual document can be incorporated into only one document series -- it cannot be incorporated into different document series simultaneously.
Add Comments
To add comments to a document, click in the Flag column.The Review panel will appear. Enter the comment and click Ok.Comments are displayed in the Document Versions panel as notes.
A summary of replicates contained in a Skyline document is available by clicking the "replicates" link in the Document Summary panel.The Replicate List provides annotation details and sample information for each replicate.For graphical views that incorporate replicate data, see the following documentation on PanoramaWeb:
Skyline can extract and store chromatogram-style data that is scoped to a raw data file. Examples include keeping track of the pressure or voltage traces recorded by the instrument during its data acquisition. This topic describes how to access the sample-file scoped chromatogram data within Panorama.
Chromatogram Data Storage
Users import raw data files as normal into Skyline. Skyline automatically inspects them and extracts whatever pressure traces or other time-series data is available, and stores it in its SKYD chromatogram file format. When these Skyline files are imported into Panorama, this information is stored in the targetedms.SampleFileChromInfo table. No additional user action is needed to include the chromatogram data.
User Interface Access
Users will be able to go to the replicate list for a given Skyline document and click to see details, including the plots of the time-series data.
Open the Skyline file of interest.
In the Document Summary, click the replicates link.
On the list of replicates, hover over the row of interest to reveal the (Details) link. Click it to open the details page.
The plots for each series of data present in the imported file will be shown below the Sample File Summary which provides about the sample and the file it came from.
API Access
The targetedms.SampleFileChromInfo table is exposed for API-based access. Developers can view the table by selecting > Developer Links > Schema Browser.
Targeted MS Chromatogram Crawler
Administrators can obtain a listing of chromatograms stored on their site via a crawler which will check each container and list chromatograms found. This assists admins in locating chromatograms that can be deleted from the database.
Select > Site > Admin Console.
Under Configuration, click Targeted MS Chromatogram Crawler.
Click Start Crawl.
You will see the pipeline progress.
When the status shows as complete, click the word COMPLETE to see the crawl log.
The log will begin with key statuses, show you a per-container record of the crawling completed, and conclude with a "Crawl summary by Skyline document counts."
Crawl Results: For each container crawled, the job will log the number of chromatograms that have byte offsets in the DB (which will be zero for non-Panorama folders or Panorama folders which have no imported Skyline files, or which were all imported before we started storing byte indices). It will iterate through all of the Skyline files in the container and inspect five chromatograms (the most typical problem will be that the server can’t find a SKYD file, so there’s little benefit to check out every chromatogram in each file). It will report the status for the chromatograms it inspects:
No data available at all: the server never had a SKYD file when it loaded data.
dbOnly: DB contains bytes, but not offset so impossible to retrieve directly from file.
diskOnly: DB does not contain bytes, but does have indices and data was retrieved successfully from file.
noSkydResolved: DB contains bytes and offset, but we can't resolve the SKYD path, perhaps because the S3 bucket is no longer configured.
skydMissing: DB contains bytes and offset, but we can't find the SKYD file that's referenced.
mismatch: Both DB-based and file-based bytes available, but they do not match.
match: Both DB-based and file-based bytes available, and they do match.
skipped: we hit our limit of 5 for the current file and didn’t examine the rest.
The protein details page can be reached by clicking the name/label from the Precursor list. Depending on the contents of the Skyline file, you will some or all of the features described in this topic. Users have the option to see multiple peptides or small molecules in the same chromatogram.After loading your Skyline document into Panorama, click the name of the file to open the details page. Then click the name of a protein / label to open the Protein/Molecule List Details page.
The Precursor List shows summary information about the Peptides and Proteins contained in the Skyline file. Use the and to expand/collapse the Peptide section. Click the name in the first text column to see more detail.Hover over the next to the name for a tooltip with more details about the sequence.
Protein Highlights
The top section of any single protein detail page shows various highlights, depending on the details present in the Skyline file.
If the Skyline file includes Protein Grouping, in which one or more peptides match not to a single protein sequence, but multiple proteins, the precursor list will show the group of proteins. The protein group label is typically the concatenated and slash-separated list of protein names, but this is not required.When you click the name of a protein group, the details page will show a highlights section for the group, then list all of the proteins in a grid. If there are protein sequences available, Panorama will automatically show the first protein's sequence (as for a standalone protein). Users can click on any of the other protein names to see the details page for that sequence.
Sequence Coverage
The Sequence Coverage map provides a heatmap style view giving the user an immediate visual assessment of:
Peptide XIC intensity
Peptide ID confidence data
On the map, color correlates intensity or confidence score, with the rank displayed on each peptide box in both views. The map helps you visualize the spatial location within the sequence. The legend maps colors to the values they represent. Hovering over any peptide box will give more details, including a grid of multiple modified forms of the peptide if they exist. The Intensity view might look like this:Use the View Settings to control the display:
By:
Intensity (shown above): Use the Log 10 based intensity value
Confidence Score (shown below)
Replicate:
All: (Default) Displays the maximum intensity or confidence of the peptide across all replicates.
Individual replicates present
Modified forms: In cases where there is more than one modified version of a peptide (modified/unmodified, multiple modifications), select how to display them:
Combined: Use the sum of intensities of all the modified forms and show in a single box.
Stacked: Show a 'stacked' set of boxes for each of the modified forms, with the modification with the highest intensity shown on top.
The Confidence Score view of another sequence might look like this:When there is no intensity data, or no confidence scores are present in the file, there will be a 'non-heatmap' version shown. For example:
Multiple Peptides Per Chromatogram
When the protein contains many peptides, it can be useful to see the chromatograms for several peptides in one plot.On the protein details page, scroll down to the Chromatograms section where you will a legend showing the colors assigned to the individual peptides present. By default all peptides are included, unless you select a subset using the checkboxes. The chromatograms below all use this scheme to show the same series of peptides for each replicate.Use the Row Size menu to control the display size of the SVG plots; options are 1, 2, 3, 4, 5, or 10 per row.
Select Peptides for Chromatograms
By default, all peptides will be included in the chromatograms, unless you use the checkboxes in the Peptides panel to select only a subset.You can also use the Display Chart Settings to to multi-select the specific replicates for which you want to see the chromatograms, making it easier to compare a smaller number of interest.Learn more about Chromatogram Display Settings in this topic:
Within Skyline, users can create lists which have one or more columns of different data types. This topic describes how to access these lists within Panorama.
Skyline Lists
When a Skyline Document import includes lists, you can access them by clicking the "lists" link in the Document Summary panel.Click the name of any list to see the contents. Annotations within the Skyline document that reference the given list will resolve as a lookups in LabKey to the relevant row.These lists are functionally similar to LabKey Server lists, but stored, scoped, and surfaced in different ways. Skyline lists are saved in Skyline's XML file format, then parsed and imported to Panorama specific tables. They can be viewed, queried, and exported from LabKey in the targetedmslists schema. Skyline lists are read-only inside of LabKey, like other Skyline data, so you will need to make any changes in Skyline and upload a new version.Skyline lists are scoped to the specific Skyline document which includes them. If the document is deleted from LabKey, the list will also be deleted.Multiple Skyline documents could each have a list with the same name without raising conflicts since the name is stored with the unique RunId as a prefix. The superset of all lists with the same name is stored with the prefix "All_". For example, if runs 106 and 107 each have a "DocumentProperties" list, the targetedmslists schema would contain:
When a Skyline document is imported, Panorama will parse and expose any annotation data, which can be used to create custom grids and visualizations.Developers can determine where annotation data (such as Condition, BioReplicate, and Run) are stored using the > Developer Links > Schema Browser. For example, peptide and small molecule annotations are stored in tables like targetedms.precursorannotation, targetedms.generalmoleculeannotation. All such tables have lookups (foreign keys) to the tables being annotated.In addition, there is an "Annotations" column on each entity (targetedms.replicate, for example). In this column, all annotations are shown as name/value pairs. Upon import, any annotation columns will also be exposed as discrete "virtual" columns on the entity table itself. Such fields can then be exposed in grids using the UI, such as in the following example using replicate annotation data.
Replicate Annotation Data
Replicate annotation data is exposed in the query targetedms.Replicate.Users can access the annotation data by customizing their data grids:
Starting from a data grid, select (Grid Views) > Customize Grid.
In the Available Fields pane, open the node Sample File and then the node Replicate.
Scroll down the Available Fields pane to see the available fields.
Select fields to add them to the data grid. The screenshot below shows how to add the Time annotation field.
Panorama will import audit logs from Skyline documents, provided audit logging is enabled in Skyline.When Panorama detects a valid audit log, the (three bars) icon is displayed in the Document Summary.
Click the document file name on the Runs tab.
In the Document Summary panel, click to see the audit log in a summary form.
In the Skyline Audit Log panel, select (Grid Views) > AllMessagesView for the even more detailed grid.
In the Document Summary panel, click the calibration curves link to see the available calibration curves.Hover over any row to reveal a (Details) button. Click to see the calibration curve plot.Calculations round to 5th decimal digit. The calibration curve plot is rendered with a regression line.Click a point in the plot to see its calculated concentration. Values are shown in the legend.Below the plot, a panel of Figures of Merit provides limit of detection and limit of quantitation details (when available). Learn more in this topic: Panorama: Figures of Merit and Pharmacokinetics (PK)
Summary Charts
Summary charts show peak areas and retention times for the replicates in a bar and box plot format.
Quantitation Ratios
Below the summary charts, a table of Quantitation Ratios includes columns for the sample file, Q1 Z, Sequence, RT, Intensity L, Intensity H, L/H Ratio, Exp Conc, and Measured Conc.
Figures of Merit for quantitation data - Summary statistics such as the mean, standard deviation, and %CV for the replicates, along with lower limit of detection, quantitation, etc.
For a given Skyline document, click the ## Calibration Curves link.
In the Peptides Calibration Curves panel, click the peptide name (or hover for a link). Click either for details for that peptide.
Scroll below the Calibration Curve on the details page to the Figures of Merit panel. Summary information, including LOD/LOQ Values, is shown here.Click Show Details for the more detailed report.The Figures of Merit report includes several sections.
The report includes the following values for both Concentrations in Standards and Concentrations in Quality Control, if present:
n - The Number of samples used for above calculations.
Mean - The mean of all replicates for a given sample.
StdDev - Standard deviation of all replicates for a given sample.
CV(%) - Coefficient of Variation for all replicates for a given sample (the standard deviation divided by the mean).
Bias(%) - The difference between expected and observed values.
Point exclusions are imported, and any excluded raw data is shown as strike-through text. Excluded values are not included in summary calculations.
Summary: LOD/LOQ Values
Within Skyline, you can set the limit of detection (LOD) calculation, the coefficient of variation (CV) limit, and limit of quantitation (LOQ) bias. These settings are picked up automatically by LabKey and included in the Figures of Merit report in Panorama.Key terms:
Limit of detection (LOD) - the lowest quantity of a substance that can be distinguished from the absence of that substance
Limit of quantitation (LOQ) - the lowest concentration at which the analyte can not only be reliably detected but at which some predefined goals for bias and imprecision are met. It is typically higher than the LOD. Note that LOQ can come in lower (LLOQ) and upper (ULOQ) varieties.
Bias - difference between expected and observed values, often represented as a percent. If no value is provided for either bias or CV, the bias cutoff will default to 30% and the CV will be ignored.
Coefficient of variation (CV) - ratio of the standard deviation to the mean.
To set these values in Skyline, open Settings > Peptide Settings > Quantification tab.Enter:
Max LOQ bias: Maximum (percentage) allowed difference between the actual internal value and the calculated value from the calibration curve.
Max LOQ CV: Maximum allowed coefficent of variation of the observed peak areas of the set of internal standard replicates with a specified concentration.
Select how to Calculate LOD from the following options:
None
Blank plus 2 * SD: Base the limit of detection on the observed peak areas of the blank samples. Add twice the standard deviation.
Blank plus 3 * SD: Add three times the standard deviation to the observed peak areas of the blank samples.
Bilinear turning point: Impute the limit of detection using the turning point of bilinear fit.
Qualitative ion ratio threshold
Both limit of detection and limit of quantification results are shown in the summary Figures of Merit section below the calibration curve (as well as at the top of the details page).
Blanks
The raw data for blanks is show at the bottom of the figures of merit report. The same statistics, except bias, are displayed as in the concentrations sections.
For a given Skyline document, click the Calibration Curves link.
In the Peptides Calibration Curves panel, click PK for a given peptide.
The link is only shown if your replicates have the Dose and Time annotations.
PK Calculations Panel
For successful PK calculations, the following annotations should be present on the Replicate data as configured in Skyline:
Dose: number (Dose that was administered)
DoseUnits: text (Units for what was administered, such as "mg/kg")
Time: number (Timepoint after administration)
ROA: text (Route of Administration, typically "IM" or "IV")
SubGroup: text (Optional) Grouping of data within a single Skyline document
The Pharmacokinetic report provides a panel of PK values, and if subgroups are provided, a panel for each subgroup.Use the checkboxes to set values for the C0 and Terminal columns. These indicate which timepoints are used as the starting and ending points of the calculations. By default the earliest 3 timepoints for C0 and the last 3 timepoints for the Terminal values are selected.The Time column in the spreadsheet will be a custom replicate annotation in the Skyline document. There will also be replicate annotations for Dose, DoseUnits, ROA (Route of Administration), and SubGroup. Dose and DoseUnits may be added to one or more replicates per SubGroup. When Dose is included on more than one replicate is must be the same value. The same applies for DoseUnits and ROA.You can print the data to Excel by clicking the icon in the Statistics header.
Non-IV Route of Administration
If the ROA (Route of Administration) is any value other than “IV” the subgroup will have an input field for “non-IV C0” and a button to Recalculate.
Charts and Raw Data
Scroll down the page to see charts and raw data panels for this subgroup, followed by sets of panels for other subgroups.
Panorama provides a summary of all the data on this server acquired from each instrument. This Instrument Summary also provides automatic links between data in Experiment folders and the QC folder(s) that contain data from the same instrument, making it quick to see how the instrument was performing at the time of acquisition.
The Instruments Summary panel lets you see which instrument(s) acquired the data in a Skyline document. It is shown on the replicate page for a single Skyline document.The summary lists, for each experiment in the folder:
QCFolders: If any QC folders exist for the instruments and experiment dates, there will be one or more links here to go to the folder, where you will see a combined view of experiment and qc plot data.
Click a link in the Serial Number column to see all data on this server acquired from this instrument. For example, clicking the "Exactive Series slot #2384" link in the image above:
Linked Experiment and QC Folders
A Panorama folder of type "Experimental Data" provides a repository of Skyline documents, useful for collaborating, sharing and searching across multiple experiments.A Panorama folder of type "QC" contains quality control metrics of reagents and instruments.When there is QC data for the instruments used in the experiments, users will find it helpful to navigate directly to the QC data around the time of experimental data acquisition to assess the performance of the instrument at the time.
Navigate the Combined Information
The experiment date range is shown in blue. Any guide sets applied or in the same time range will be shown in gray (overlapping sections will be a mixed darker shade).In the blue bar above the plot, click Return to Experiment to return to the experiment folder.
Using Panorama, you can import, parse, and view small molecule data, such as lipids or other non-amino-acid compounds, inside of Skyline documents. Skyline documents containing mixed sets of peptides and small molecules can be imported. Panorama will separate the mixed peptides and small molecules into their respective views.Views provided inside Panorama include:
Small Molecule Precursor List
Small Molecule Summaries
Small Molecule Details, including Chromatograms
All of these views are similar to the analogous peptide views, though spectrum graphs are not shown for small molecules.
Click the Data Pipeline tab. In the Data Pipeline web part, click Process and Import Data.
Drag and drop the individual Skyline documents or .zip file into the Files web part. This walkthrough uses sample test data; your column names and results may vary.
When the documents have been uploaded, select the documents and click Import Data.
In the Import Data popup menu, confirm that Import Skyline Results is selected, and click Import.
When the import job is complete, click the Panorama Dashboard tab.
In the Targeted MS Runs web part, click a Skyline document for views and details on the contents.
Available Views
The Small Molecule Precursor List shows a summary of the document contents.
Under Molecule List you will see sections for each molecule group. In the above screenshot, "Library Molecules" is circled and shown expanded.
The expanded grid column names may vary. In this screenshot, under Molecule you see links (such as "Cer(d14:1/22:0)"). Click for a details page.
Click the molecule group name, here Library Molecules to see a summary page which includes charts showing peak area and retention time information.
Click the small PDF and PNG icons next to the plot titles to export to either format.
Go back to the Document Summary. Clicking the ## transitions link will switch the page to show you the Small Molecule Transitions List.
Ion Details
The following screen shot shows details and chromatograms for an ion, here "Cer(d14:1/22:0)" clicked from the Small Molecule Precursor List.
Related Topics
Use a Panorama QC folder to track the performance of instruments and reagents using Levey-Jennings plots, Pareto plots, and other tools for both proteomics and small molecule data.
Heat maps are a powerful way to visualize expression matrix data. Clustergrammer is a free visualization service and open source project provided by the Ma'ayan Lab at the Icahn School of Medicine at Mount Sinai. A heat map for data from runs in Panorama can be generated using the free web service version of Clustergrammer.
Generate a Clustergrammer Heat Map
Navigate to the Panorama runs list of interest.
Select the runs of interest and click Clustergrammer Heatmap.
Adjust the auto-generated title and description if desired, or accept the defaults.
Click Save.
You'll be asked to confirm that you consent to publish the information to Clustergrammer. Clustergrammer is a third-party service, so all data sent will be publicly accessible. Click Yes if you wish to continue.
The heat map will be generated and shown; depending on your run content, it might look something like this:
When you generate a Clustergrammer heat map, the server auto-generates a Link Report giving you a way to access it later. To see the link report, add a Data Views web part:
Enter > Page Admin Mode.
Select Data Views from the Add Web Part dropdown in the lower left.
Multi-Attribute Method (MAM) analysis improves simultaneous detection, identification, quantitation, and quality control (monitoring) of molecular attributes. Panorama includes MAM-specific reporting with its analytics, offering immediate results and data sharing to any imported data. Additionally, Panorama and AutoQC’s automated workflow provide longitudinal tracking of MAM-related metrics for QC purposes.
Poster
A poster detailing how Panorama features support for MAM-related reports was presented at the ASMS (American Society for Mass Spectrometry) Conference in 2020:
The Multi-Attribute Method folder is the same as the Experiment Data folder with additional reports provided.To set one up:
Create a new project or folder, choosing folder type Panorama.
On the Configure Panorama Folder page, select Multi-Attribute method (MAM).
You can also change an existing Experiment folder to add the MAM reports by going to > Folder > Management > Module Properties and setting TargetedMS Folder Type to "ExperimentMAM" for the folder.
MAM Reports
You'll see links to the additional reports in the Document Summary section.Learn more about the Multi-attribute Method Reports in these topics:
The Post-translational Modification (PTM) Report shows the proportion for each peptide variant's peak area across samples. Panorama automatically groups peptides with identical unmodified sequences, but a user can configure alternative groupings within the Skyline document, to group splice variants, for example.
Early Stage PTM Report
Users doing early-stage antibody development want an efficient, automated way to assess the stability of their molecules and characterize the risk for each potential modification based on sample and protein criteria. The Early Stage PTM Report addresses this need with the following key features:
Stressed vs DS/FDS is a property of the sample, and captured in a Skyline replicate annotation, "Stressed or Non-Stressed"
CDR vs Non-CDR is based on whether the modification sits in a "complementarity-determining region"
These are the sections of the protein that are most important for its ability to bind to antigens
They will be specified as a list of start/end amino acid indices within the protein sequence via a "CDR Range" annotation in the Skyline document, attached to the protein (aka peptide group). Example value for a protein with three CDRs: [[14,19],[45,52],[104,123]]
CDR sequences within the overall peptide sequence are highlighted in bold.
A Max Percent Modified column represents the maximum total across all of the samples. This lets the user filter out extremely low abundence modifications (like those that have < 1% modification)
Two pivot columns are shown for each sample in the Skyline document (shown as split columns). The headers show the value of the "Sample Description" replicate annotation, when present. Otherwise, they use the sample name.
Percent Modified: Modification %, matching the value in the existing PTM report
Total Percent Modified: Total modification %, which is the sum of all modifications at the same location (amino acid)
When there is a single modification, it's the same value
When there are multiple modifications, the "shared" cells span multiple rows
Special case - for C-term modifications the value to be shown in the % that is NOT modified. That is, 100% - the modified percentage
Highlighting in red/yellow/green is applied based on whether the peptide is or is not part of a CDR sequence and whether it is observed in a stressed or non-stressed sample.
Peptide Map Report
The Peptide Map Report shows the observed peptides in the MAM analysis, sorted by retention time.
It compares the observed and expected masses, and the modifications for each peptide variant.
Shows the protein chain and the peptide location as two separate columns instead of combining into a single column.
Includes the next and previous amino acids in the Peptide column (each in parentheses).
Shows the location of modifications on the protein/peptide sequence in addition to just the modification name.
To support multi-attribute method analysis, Skyline supports crosslinked peptides as special peptide modification annotations. When crosslinked peptides are included in a Skyline document imported to Panorama, the foldershow all of the peptides involved in the linking and highlight the amino acids involved. Crosslinked peptides sequences are parsed into a reusable data structure, and link locations are rendered as modifications.
Crosslinked Peptides
Peptide crosslinks are a form of post-translational modifications for peptides. Peptides are connected via a small linking molecule. One or more other peptides could be linked from each amino acid.When Panorama renders these peptides and their precursors, we now show all of the peptides that are involved in the complex, each on a separate line. The amino acids involved in the linking are highlighted in green.
Peptide Map Report
When cross-linked peptides are present, the peptide mapping report for MAM folders will show details about how and where the peptides are linked.
Chromatogram libraries capture representative chromatograms for a given target in a given set of instrumentation. Users can upload additional documents and then curate which version of a chromatogram is considered the most representative and should be the one stored in the library. A completed library can be downloaded as a .clib file, which is a SQLite database. That can then be imported into Skyline to inform and score future assays that use the same small molecules, peptides, or proteins.Choose the Chromatogram Library folder type to include the tools for creating libraries. The Rank peptides within proteins by peak area box should be checked when you have targeted results containing relative peptide peak areas for a given protein. Leave this box unchecked if you do not yet have quantitative data from proteolytically digested proteins, such as when you are working with synthetic peptides.
Tutorial
The Panorama Chromatogram Libraries Tutorial shows you how to build a chromatogram library for peptides. The same type of functionality is available for small molecules. In this tutorial you will:
Build a chromatogram library in Panorama
Use it in Skyline to select peptides and product ions to measure in a new experimental setting
Compare library chromatograms with new data to validate peptide indentifications.
The Panorama Chromatogram Libraries Tutorial shows you how to build a chromatogram library for peptides. The same type of functionality is available for small molecules, as shown in the example on this page.Highlights and attributes for small molecules include ion-mobility types of data like cross-sectional profiles and mobility, collision energy, and more.
Obtain your Skyline file. Within Skyline, you will publish to Panorama, or you can use the Data Pipeline to import within Panorama. You can find a few examples in the LabKey test data on github if you want to demo these tools before you have your own data.In this walkthrough, we start with the "SmMolLibA.sky.zip" and later import the "SmMolLibB.sky.zip" file to show how to resolve conflicts. While this example shows small molecule data, the features are by no means limited to that type of data.
Create a new folder of type Panorama, selecting Chromatogram library.
Click the Data Pipeline tab.
Click Process and Import Data.
Drop your Skyline file(s) into the target area to upload them.
Select the desired file, then click Import Data.
In the popup, confirm that Import Skyline Results is selected, then click Import.
You will see import progress in the pipeline. When complete, continue.
All Skyline files uploaded to a library folder will be eligible to include in the .clib file. Whether or not the user has to resolve conflicts, Panorama will choose the molecules based on their Q values, if available. Panorama will choose the one with the minimum Q value as the reference trace. If Q values do not exist, Panorama will then pick the one with the most intense peak. Options for the user to pick which trace to use in the library will not be supported in this iteration.The Quantitative/non-quantitative flag for transitions shall be added to the .clib file.
See Summary Data
Click the Panorama Dashboard tab to see the overview of the data available in this library. Note the Molecules tab on the right, shown when small molecule data is present.
Note that if you checked the box to "Rank peptides within proteins by peak area" you will see a different dashboard combining information about proteins and peptides with your small molecule data. If you see the tabs Peptides and Proteins, you may want to recreate the folder without that box checked to work with small molecule data.
Tour Page Layouts and Folder Tabs
As shown above, there are three web parts:
Chromatogram Library Download: Summary information, statistics, and a button to download this revision.
Mass Spec Search: Search for proteins, peptide sequences, or peptides with specific modifications.
Targeted MS Runs: This panel lists the Skyline files you have imported, with the counts of molecule lists, small molecules, precursors, transitions, and replicates.
Click the Molecules tab for lists of molecules and precursors.
View Details and Chromatograms
Click the name of a molecule in the Molecules webpart, Molecule column (shown here "C4H9NO3") for a summary, listing of precursors, chromatograms, and panel for generating summary charts at the bottom of the page.Learn about chromatogram settings and options in this topic:
If documents are imported with conflicting information, the dashboard will show a message indicating the number of conflicting molecules. You must resolve all conflicts before you can extend this library. Click Resolve Conflicts to do so.For each conflict, you will see the Newly Imported Data on the left and the Current Library Data on the right. In each row, select the checkbox for the version you want to include in the library. Click Apply Changes when finished.
Download .clib SQLite Files
To download the .clib SQLite files, click Download on the Panorama Dashboard.You can import this .clib SQLite file into Skyline and use it to score future experiments.Note that if you download the library while there are unresolved conflicts, you will download the latest stable version of the library (i.e. it may be missing more recently added information).
To support the development of robust and reproducible assays, Panorama offers reporting that summarizing analysis of replicates of the same sample multiple times a day over multiple days to assess inter- and intra-day variability. 3 replicates over 3 days or 5 replicates over 5 days are common usages. As long as at least 3 runs are included, Panorama will generate the Reproducibility report for that protein. The report relies on a Replicate annotation in Skyline to identify the day for grouping purposes. The Reproducibility Report visualizes the reproducibility of the assay and which peptides perform best.The current implementation of the report is specific to protein/peptide data and does not yet support small molecule data.
For this report, you will load a Skyline document containing multiple replicates for a protein. You must include a Day replicate annotation, and you may want to also captures both NormalizedArea and CalibratedArea values, in order to take advantage of the options available below.To configure, start in Skyline:
Select Settings > Document Settings
On the Annotations tab, click Add...
Use "Day" as the name, and leave the Editable button toggled
Check Replicates from the "Applies to" list
View->Document Grid and choose "Replicates" from the Reports drop-down
Fill in values for the Day column for the replicates that you want to include in the reporting. You can use whatever values you like to identify the timepoint - "Day 1", "Day 2", "10-08-2021", "Batch 1", etc.
Optionally, add annotations to capture the normalized and calibrated areas.
Select Settings > Document Settings
On the Annotations tab, click Add...
Use "NormalizedArea" or "CalibratedArea" as the name, and choose the Calculated button
Check Precursor Results from the Applies to list
Choose the value to plug in for the annotation. Precursor->Precursor Results->Precursor Quantitation->Precursor Normalized Area or ->Calculated Concentration. Use "Sum" as the aggregate
Create Folder and Load Document
To use this report, start in a Panorama folder of type "Chromatogram library".When you import a Skyline document containing protein data, the system will automatically add a new Proteins tab.The protein list on this tab contains Protein/Label, Accession, Preferred Name, Gene, Description, Species, File Name, Protein Note/Annotations and Library State (if available). For proteins where there are multiple replicates included in the document, you will see a link next to the protein / label. Click it to open the Reproducibility Report.
View Reproducibility Report
The report will look for a Replicate annotation named "Day" in the Skyline document to analyze for intra- and inter-day variability. In the absence of this annotation, the report will not be available.The report contains a series of panels:
Protein: This overview section shows some basic metadata about the protein and the file it was loaded from.
Sequence Coverage(if available): View the peptide coverage within the protein sequence. Select among viewing options.
Precursors: The precursor listing can be filtered and sorted to customize the report.
If the Skyline document includes paired heavy and light precursors, you will be able to select either Intensity or Light/Heavy Ratio values for the report. Learn more below.
Any CV values (inter-day, intra-day, or total) over 20 in this section are highlighted in red.
You can click the link in the second column header to Copy the filtered peptide list to the clipboard.
Depending on the selection in the Precursors panel, the first plot is one of these:
Chromatograms: Clicking on a precursor in the plots or summary table will select that precursor and scroll to its chromatograms at the bottom of the page.
Calibration Curve/Figures of Merit: If the imported Skyline file contains calibration data, we will show the Calibration curve on the page. Figures of Merit (FOM) will be displayed below the calibration curve.
The Figures of Merit section will only display the Summary table, with the title linking to the detailed FOM page
The Calibration Curve title is also a link that would bring the user to a separate Calibration Curve page.
Annotations
If relevant annotations are available, they will be shown in the next panel. For example:
Precursor Listing Options
If the Skyline document includes paired heavy and light precursors, you will have two ways to view the reproducibility report, selectable using the radio button in the Precursors web part. This selection will determine the plots you will see on the page.
Intensity: Sort and compare intensity for peak areas and CV of peak areas data.
Light/Heavy Ratio: Instead of peak area, analyze the ratio of light/heavy precursor peak area pairs (typically performed by isotope labeling).
Sort by lets the user sort the precursors on the plots shown by:
Intensity or Light/Heavy Ratio, depending on the radio button selection.
Sequence
Sequence Location (if two or more peptides are present)
Coefficient of Variation
Value type: When the document contains calibrated and/or normalized values, you can select among them. The default will depend on what's in the Skyline file: Calibrated, Normalized, and Raw in that order.
Calibrated
Normalized
Raw
Sequence Length: Drag the min/max endpoint sliders to filter by sequence length.
Click the link in the second column header to Copy the filtered peptide list to the clipboard.
Intensity (Peak Areas)
When the Intensity radio button is selected in the Precursors web part, the data grid will show:
Peptide sequence
Charge
mZ
Start Index
Length
Inter-day CV (Note that if the report includes 1xN (a single day), the inter-day CV column will still be present, but values will be 0.0%.)
Intra-day CV
Total CV
Mean Intensity
Max Intensity
Min Intensity
The Intensity web part will plot the peak areas with the selected filters and sorts.The plot will show each replicate's peak area, divided by run within the day, and day. A box plot will show the full range of all values. Data points will be arranged from left to right within each precursor based on their acquisition time.
Intensity Coefficient of Variation
Based on which Show checkboxes are checked, this plot shows the averages for intra-day and inter-day peak areas, plus the "Total CV", which is the square root of the sum of the squares of the intra and inter-day CVs.On the CV plot, hover over individual bars for a tooltip:
For the inter- and intra-day CVs, see the full peptide sequence (with charge state and mz) and the standard deviation of the CVs, as well as the average CV (which is the value being plotted).
For the total CV, you'll see the value and peptide sequence.
Light/Heavy Ratios
If the Skyline document includes paired heavy and light precursors, and Light/Heavy Ratio is selected, the data grid will contain the following information:
Peptide sequence
Charge (light and heavy)
mZ (light and heavy)
Start Index
Length
Inter-day CV of light/heavy ratios of intensity (Inter-day CV)
Intra-day CV of light/heavy ratios of intensity (Intra-day CV)
Total CV of light/heavy ratios of intensity (Total CV)
Mean value of light/heavy ratios (Mean Ratio)
Max value of light/heavy ratios (Max Ratio)
Min value of light/heavy ratios (Min Ratio)
The CV values will be highlighted in red if the number exceeds 20%. When a heavy ion is not present, the light/heavy ratio for that ion cannot be calculated. In this edge case, the Light/Heavy ratio for this ion will be displayed as "N/A" in the data grid. These ions will also not be shown in the Peak Areas Ratio and CV ratio plots.Light and heavy ions of the same type will be shown as one type of ion on the chart. For example: ratio data for EAEDL[+7.0]QVGQVE and EAEDLQVGQVE will be shown as EAEDL...+. If the light and heavy ions have different charges, then the ion will be shown on the graph without the charge information.
Light/Heavy Ratio Coefficient of Variation
Based on which Show checkboxes are checked, this plot shows the averages for intra-day and inter-day peak areas, plus the "Total CV", which is the square root of the sum of the squares of the intra and inter-day CVs.
Chromatograms
One chromatogram per replicate is rendered, in a matrix-like layout, with rows that represent the days the data was acquired and columns that represent the indices within the day. Hence, the layout will reflect the MxN nature of the data.When showing light/heavy ratios, the chromatograms at the bottom of the page will show both the light and heavy precursors plotted together for each replicate.To see the full layout, expand your browser view. Individual plots are titled based on the day and index.
Users can click to download a PDF or PNG of any chromatogram. Hover to see the buttons.
The Panorama QC folder is designed to help labs perform QC of their instruments and reagents over time. Runs are uploaded using the data pipeline or directly from Skyline. Panorama QC folders support reviewing and plotting both proteomics (peptide/protein) and small molecule data.
Users can copy a Panorama QC folder to a new location, either on the same server or on a different one, preserving all QC and System Suitability data to demonstrate that the instrumentation was performing well and the experimental data is trustworthy. To make the copy, you can export to an archive and reimport to another server of the same (or higher) version. All the metrics, view settings, and annotations related to Panorama QC folders are included, including but not limited to:
Guide sets and exclusions
Annotations and QC annotation types
Custom metrics and trace metrics, with their configurations and settings
Default view settings for QC plots
To copy a Panorama QC folder:
Navigate to the folder to copy.
Select > Folder > Management.
Click the Export tab.
Confirm that Panorama QC Folder Settings is selected, along with other desired objects to include in the archive.
Select the Export to option; the default is "Browser as zip file".
Click Export.
Now create a new folder on the destination server.
The Panorama QC Overview dashboard offers a consolidated way to view good summaries of system suitability and quality control information. Information from the current folder and immediate subfolders are displayed in a tiled format. For example, subfolders might each represent a specific machine, so you can see instrument conditions at a glance.The Panorama Dashboard tab shows the QC Summary and QC Plots web parts.
QC Summary
On the Panorama Dashboard, the QC Summary section shows a tile for the current folder and each immediate subfolder the user can access. Typically a folder (tile) would represent an individual instrument, and the dashboard gives an operator an easy way to immediately scan all the machines for status. Each tile includes a visual indicator of AutoQC status.The count of the number of Skyline documents (targetedms.runs) and sample files (targetedms.samplefile) are scoped to the tile's container. Each tile lists the number of files that have been uploaded, the number of precursors that are being tracked, and summary details for the last 3 sample files uploaded, including their acquired date and any outliers that were identified. Each tile includes a link to View all ## samples and utilization calendar .
Learn more about the symbols used for indicating outliers below.
The TargetedMS module uses the AutoQC tool to ping the server to check if a given container exists, evaluate LC MS/MS performance over time, and quickly identify potential issues.
Using AutoQC ResultsThe QC Summary web part displays a color indicator:
Gray ring - AutoQC has never pinged.
Red circle - AutoQC has pinged, but not recently.
Green check - AutoQC has pinged recently. The default timeout for "recently" is 15 minutes.
Hover over the icon for the time of the last ping.AutoQC also facilitates the ongoing collection of data in a Panorama QC folder. Each sample is uploaded incrementally and automatically by AutoQC. AutoQC adds new samples to an existing Skyline document, rotating out and archiving old data to prevent the active file from getting too large. As new files are received, they are automatically coalesced to prevent storing redundant copies across multiple files. Whenever a PanoramaQC file receives a new file to import, by the end of that import we have the most recent copy of the data for each sample contained in the file, even if it had been previously imported.
Outlier Indicators
When outliers are present, the type of outlier determines which indicator is used to represent the severity of the issue. If a file contains more than one type of error, the 'highest' indicator is shown.
Error: A Levey-Jennings outlier
Warning: A Moving Range outlier
Info: A CUSUM outlier
: No outliers
: File excluded
Sample File Details
The display tile shows the acquired date/time for the latest 3 sample files along with indicators of which QC metrics have outliers in the Levey-Jennings report, if any.
Hover over the icon for a sample file in the QC Summary web part to see popover details about that file.The hover details for a sample file with outliers show the per-metric "out of guide set range" information with links to view the Levey-Jennings plot for that container and metric.
Delete a Sample File
To delete an unwanted sample file, such as one you imported accidentally, click the link showing the number of sample files in the folder to open a grid, select the relevant row, and click Delete. The data from that sample file will be removed from the plot.
QC Plots
The QC Plots web part shows one graph per precursor for a selected metric and date range. Choose from a variety of different available Plot Types, picking a Date Range, Scale, and other options. For more details, see Panorama QC Plots.Example Plots with guide sets and annotations:
Configure Included and Excluded Precursors
The precursors, whether peptides or small molecules, can be divided into two groups for mass spectrometry analysis:
Included group: The precursors driving the plots and AutoQC assessments.
Excluded group: Those you want to exclude from AutoQC assessments and from plots (unless you check the box to "Show Excluded Precursors").
From the QC Summary or QC Plost web part, choose Configure Included and Excluded Precursors from the menu.You'll see a grid of the precursors available in the files, with some summary data. By default, all are marked as "Included". Select one or more precursors using the checkboxes, then move them between the two groups by clicking either Mark as Included or Mark as Excluded.In the QC Plots web part, the Show Excluded Precursors checkbox controls whether to show those marked as being in the "Excluded group".
In the QC Summary panel on the Panorama QC Folder dashboard, each instrument tile includes a link to View all ## samples and utilization calendar .The QC Summary History page shows the Utilization Calendar, a heat map showing days with more of the selected source as darker. There is a dynamic key along the bottom edge indicating what the current scale is. Any days on which the instrument has been marked as Offline will be shown with a gray shade across half the box.You can choose to display 1, 4, or 12 months of data in the heat map. The default is 1 month. Choose the Heat map data source:
Sample count
Median outliers
Step through more months using the and icons on the ends of the month-title bar.An example to explore on PanoramaWeb:
Hover over a cell to see the status as well as the samples and/or outliers that the shading represents.
Mark Days as Offline
A user with the "Editor" role (or higher) can click (or click-drag to multiselect) in order to mark selected days as offline. Manually annotating an instrument as being offline, whether or not it acquired samples on a particular day, helps track instrument utilization and availability.Enter a Description and Date, as well as an End Date for a series of days.When there is data available on an 'offline' day, the gray shading will be on top of the heatmap color. Hover over any offline day to see the comment that was included, as well as a listing of the samples/outliers.To remove an offline annotation, click it again and there will be a Delete button to remove it.
The Panorama QC folder is designed to help labs perform QC of their instruments and reagents over time. Runs are uploaded using the data pipeline or imported directly from Skyline. QC Plots, covered in this topic, give a visual illustration of performance over time that can make data validation easier and more reliable.
A given molecule will have many potential precursors. You want to choose the one that gives a robust, consistent signal. In practice, for a given set of instrumentation, you'll wind up being able to best measure certain charge states, and hence, certain precursors of a given molecule.The QC Plots web part shows one graph per precursor by default. The web part header allows you to specify a number of options, including selecting one or more Plot Types to include. Hover over a plot name on the dropdown menu to learn more about it.
QC Plot Types
Metric Value: (Default) A metric value plot shows the raw value of the metric. It may be compared against fixed upper and lower bounds to identify outliers, or use a Levey-Jennings-style comparison based on the number of standard deviations (SD) it differs from the metric's mean value.
Moving Range (MR): Plots the moving range over time to monitor process variation for individual observations by using the sequential differences between two successive values as a measure of dispersion.
CUSUMm: A CUSUM plot is a time-weighted control plot that displays the cumulative sums of the deviations of each sample value from the target value. CUSUMm (mean CUSUM) plots two types of CUSUM statistics: one for positive mean shifts and the other for negative mean shifts.
CUSUMv: A CUSUM plot is a time-weighted control plot that displays the cumulative sums of the deviations of each sample value from the target value. The CUSUMv (variability or scale CUSUM) plots two types of CUSUM statistics: one for positive variability shifts and the other for negative variability shifts. Variability is a transformed standardized normal quantity which is sensitive to variability changes.
Trailing CV: A Trailing Coefficient of Variation plot shows the moving average of percent coefficient of variation for the previous N runs, as defined by the user. It is useful for finding long-term trends otherwise disguised by fluctuations caused by outliers.
Trailing Mean: A Trailing Mean plot shows the moving average of the previous N runs, as defined by the user. It is useful for finding long-term trends otherwise disguised by fluctuations caused by outliers.
QC Plot Features
Select the metric to plot using the pulldown menu in the web part header. Set other plot options to control the display of plots in the web part.
Metric: Select the desired metric from the pulldown. Learn more below.
Date Range: Default is "Last 180 Days", relative to the most recent sample in the folder (as opposed to relative to "today"). Other options range from last 7 days to the last year, or you can choose "all dates" or specify a custom date range.
Plot Types: Select one or more options on the dropdown menu for plot types to show. Options outlined above.
Y-Axis Scale: Options: linear, log, percent of mean, or standard deviations. Not all are applicable to every plot type. See below.
Group X-Axis Values by Date: Check this box to scale acquisition times based on the actual dates. When this box is not checked, acquisition times are spaced equally, and multiple acquisitions for the same date will be shown as distinct points.
Show All Series in Single Plot: Check this box to show all fragments in one plot.
Show Excluded Samples: Check to include points for samples you've excluded from your plot(s) as outliers or for other reasons. Excluded points will be shown as open circles.
Show Reference Guide Set: Always include the guide set in the plot, even if it is outside the range that would otherwise be shown.
Show Excluded Precursors: Check to include precursors (peptides or small molecules) that are marked as in the "Excluded group". By default all are in the "Included group". Note that the "Excluded group" is not related to sample exclusions.
Hover Details
Hovering over a point in the graph will open a tool tip with additional information about that data point. When the plot includes multiple precursors, such as when the "Show all series in a single plot" box is checked, the plot will also be 'filtered' to show only the line(s) associated with that precursor.From the hover tooltip, you can click to view either the chromatogram or sample file for this point. In the Status section of the tooltip, you can select an exclusion option if needed, such as for an outlier data point. Elect either of the following options to do so:
Exclude replicate for this metric
Exclude replicate for all metrics
Outliers and Exclusions
Outlier points, those outside the set metric thresholds, are shown as a triangle on plots, where other points use a circle.Excluded samples are left out of the guide set range calculations and QC outlier counts for the QC summary panel and Pareto plots. When excluded points are included in the plots, such as by checking the box "Show excluded samples", they will be shown as an open circle on the plot, where included points are solid.Outlier samples can also be excluded from QC by setting an "ignore_in_QC" replicate annotation to "true" within Skyline. This will cause the sample to be excluded for all metrics.When a whole metric has been excluded for a given sample, the QC dashboard hover information will note "not included in QC".Note that when using the Trailing CV or Trailing Mean plot type(s), and checking the Show Excluded Samples box, the trailing calculation will include the excluded samples. Data dots on the trailing plot will be a solid dot in this case, as the trailing calculation is not an excluded point causing the 'open circle' styling.
Plots with Large Numbers of Points
When the plot includes with a large number of data points per series, individual points would be too bunched together to make finding the one you are looking for practical to hover over.
The trend line for plots showing over 300 points will not render the individual points.
Hovering over the line will show the message that narrowing the date range might let you see individual data points.
Filter by Replicate Annotations
If the Skyline document you imported included replicate annotations, you will have the option to filter your plots using the Filter: Replicate Annotations drop down that will be present.Check the box(es) for the annotation(s) to select and click Apply. Selected annotations will be listed in the panel.Click Clear to remove the filtering.
Y-Axis Scale Options
All plot types support the selection of Linear or Log scale for the Y-axis. The type selected will be shown along the left sidebar, including a units label for non-CUSUM plots. This labelling does not apply to Transition/Precursor Area plots which use other labelling on the dual Y-axes.For Metric Value and Moving Range plots, you can also normalize the plots using two additional Y-axis scale types. Note that if you select these scales for CUSUM plots, a linear scale is used instead.
Percent of Mean:
The mean is considered as 100% and other values are plotted above or below based on the percent over or under the mean they are.
Value plotted as (value/mean) * 100%.
Metric Value plots will shown the bounds of +/- three standard deviations over and under the mean.
Moving Range plots will include upper and lower limits as a percent of mean, showing the variability.
Standard Deviations:
The mean is plotted as zero on the Y-axis. Values are plotted based on the difference from the mean as a ratio with the standard deviation.
Metric Value plots include standard deviation bars of +/- three standard deviations.
For example, this plot of Retention Time has been normalized to percent of mean.The range shown along the Y-axis is based on the variance of the values. This means the size of the range will be dynamic to fit all normalized data points. Metric Value plots include a default guide set to cover the entire plot when viewing a single series in a single plot.
Metrics
Administrators have the ability to control which metrics are available for plotting, as described in this topic:
There are several isotopologue metrics that will be available when there is data they can use:
Isotopologue Accuracy
Isotopologue LOD: Limit of Detection
Isotopologue LOQ: Limit of Quantitation
Isotopologue Regression RSquared: Correlation coefficient calculated from the calibration curve
Isotopologues are molecules that differ from each other solely by their isotopic composition. They are valuable in mass spectrometry because it’s possible to create variants of the same peptide with different masses. These isotopologues can then be added at different concentrations in a single sample, allowing for a calibration curve for a peptide in a single sample (and therefore single run of the mass spec). Other common calibration curve approaches require separate samples for each concentration of a single peptide.Researchers can use these isotopologue samples for QC and system suitability purposes. By monitoring the limit of quantitation (LOQ), limit of detection (LOD), and other metrics, a researcher can gain insight into the performance of the instrument and the dynamic range it’s accurately measuring. Commercial kits, such as the Pierce™ LC-MS/MS System Suitability Standard (7 x 5 Mix) are an easy way to implement this kind of monitoring.Skyline provides the ability to populate attributes on data elements like precursors and peptides with calculated values. The isotopologue metrics are used in conjunction with AutoQC to provide automated system suitability monitoring in Panorama. Panorama QC will look for the following calculated annotations on the Precursor Results elements. The name of each annotation must match exactly:
PrecursorAccuracy, populated from Precursor Quantification > Precursor Accuracy
RSquared, populated from Peptide Result > Replicate Calibration Curve > Replicate R Squared
LOD, populated from Peptide Result > Batch Figures of Merit > Batch Limit of Detection
LOQ, populated from Peptide Result > Batch Figures of Merit > Batch Limit of Quantification
When present in the Skyline document, Panorama will import these values as annotations on a PrecursorChromInfo data element (captured in the targetedms.precursorchrominfo table during import) with the names “LOD”, “LOQ”, “PrecursorAccuracy”, and “RSquared”.Example data from the 7x5 kit is available on PanoramaWeb , as is a template Skyline document for analyzing raw data files from the kit.
Run-Scoped Metrics
Metrics that are not tied to an individual molecule can be tracked and plotted using metrics that produce a single value for the entire run. The built-in metric "TIC Area" (Total Ion Chromatogram Area) shows an example of such a run-scoped metric.When viewing run-scoped metrics, the "Show all Series in a Single Plot" option is not available since there is only one series to show.
Transition & Precursor Areas
Both Transition Area and Precursor Area can be plotted. To show both precursor and fragment values in the same plot, select the metric option Transition & Precursor Areas. Hover over a point on either line to confirm which axis it belongs to.The plot is more complex when all fragments are shown. Use the legend for reference, noting that long peptide names are abbreviated.Hover over a truncated name to see the full peptide name, as well as the plot filtered to show only lines associated with that peptide.
Set Default View
When an administrator has customized the QC Plots web part and metrics settings for a container, they have the option to save these settings as a default view for all other users visiting this folder. From the menu, select Save as Default View.Users viewing the plots will now see the admin-defined default plot configuration settings, unless they have customized their own view of plots in this container. Users can start from the default and make changes as they work, coming back to their own settings when the return.All users also have the option to Revert to Default View at any time. If no admin-customized default view is available, the original default settings will be applied.
Export
You can export any of the plots by hovering to expose the buttons, then clicking the icon for:
PNG: Export to a PNG image file.
PDF: Export as a PDF document.
Small Molecule Data
The same Metric Value, MR, CUSUM, and Pareto plot features apply to both proteomics (peptide/protein) data and small molecule data. Data from both types may be layered together on the same plot when displaying plots including all fragments. Counts of outliers and sample files include both types of data.When visualizing small molecule data, you are more likely to encounter warnings if the number of precursors exceeds the count that can be usefully displayed. This screenshot shows an example plot with small molecule data.Note that the legend for this illegibly-dense plot does not list all 50 precursors. You can narrow what the plot displays by marking uninteresting small molecules as in the "Excluded group" to hide them.
QC Metric Settings Persistence
Once any user has configured a useful set of metrics and ways of viewing your plots, they will be persisted. The next time that user visits this folder and the plots on this dashboard, they will see the same metric they were most recently viewing with the same settings they were using.
The default plot in a Panorama QC folder is the Metric Value plot which shows the raw value of the metric and can be helpful in visualizing and analyzing trends and outliers. The metric value may be compared against fixed upper and lower bounds to identify outliers, or use a Levey-Jennings-style comparison based on the number of standard deviations (SD) it differs from the metric's mean value.Configure metrics and outlier thresholds as described in this topic:
The moving range can be plotted over time to monitor process variation for individual observations by using the sequential differences between two successive values as a measure of dispersion. Moving Range (MR) plots can be displayed alongside Metric Value plots for integrated analysis of changes. To create a Moving Range plot, select this option from the Plot Types dropdown. Note that you can multiselect plot types here.In this screencap, both the Metric Value and Moving Range plots are shown. Notice the two elevated points on the moving range plot highlight the largest changes on the other plot. Otherwise the value for retention time remained fairly consistent in two level 'zones'.The plot configuration features outlined in Panorama QC Plots also apply to Moving Range plots.
CUSUM Plots
A Cumulative Sum (CUSUM) plot is a time-weighted control plot that displays the cumulative sums of the deviations of each sample value from the target value. This can highlight a problem when seemingly small changes combine to make a substantial difference over time.The legend for the dotted and solid lines is included on a CUSUM plot:
CUSUM- is a solid line
CUSUM+ is a dotted line
The plot configuration features outlined in Panorama QC Plots also apply to both types of CUSUM plots.
CUSUMm (Mean CUSUM)
The CUSUMm (mean CUSUM) plots two types of CUSUM statistics: one for positive mean shifts and one for negative mean shifts.
CUSUMv (Variable CUSUM)
The CUSUMv (variability or scale CUSUM) plots two types of CUSUM statistics: one for positive variability shifts and one for negative variability shifts. Variability is a transformed standardized normal quantity which is sensitive to variability changes.A sample CUSUMv plot, shown with no other plot type selected:
Trailing Mean and CV Plots
The Trailing Mean and Trailing CV plots let the user set the number of previous samples over which they want the trailing value (mean or coefficient of variation, respectively) calculated for the metric selected. These plots are useful for finding long-term trends otherwise disguised by fluctuations caused by outliers.After selecting either of the "Trailing" plots, you'll see a new Trailing Last entry box for entering the number of runs to use to calculate the value. You must enter a positive integer, and fewer than the existing number of runs, otherwise you'll see an error message. The default is 10 samples (or all samples if there are fewer than 10 total).The plot configuration features outlined in Panorama QC Plots also apply to both types of "Trailing" plots. Hover over a point for more details, including:
Peptide/Molecule: The peptide or small molecule name, charge state, and mZ value
Value: The calculated trailing value over the last N samples
Replicate: Number of samples used (N)
Acquired: Data acquisition begin and end dates for the samples included
Trailing Coefficient of Variation
A Trailing Coefficient of Variation plot shows the percent coefficient of variation for a selected metric over the previous N samples, as defined by the user.The Y-axis will show the CV(%).
Trailing Mean
A Trailing Mean plot shows the average of a selected metric over the previous N samples, as defined by the user.The Y-axis for the Trailing Mean plot will vary based on the metric selected for the plot. Some plot types don't label the Y-axis.
Annotations can be added to Panorama QC plots to assist in quality control. Marking a specific event can help identify causes of changes. If there are replicate annotations present in the imported Skyline data file, they will also be shown and can be used to filter plots.
Color coded markers can be used to annotate the QC plots with information about the timing of various changes. The annotated plot will show colored Xs noting the time there was a change in instrumentation, reagent, etc. The legend shows which color corresponds to which type of annotation.Hovering over an annotation pops up a tooltip showing information about when the annotation event occurred, the description of the event, and who added it.
Add Annotations
Select the Annotations tab to define and use annotations.
Define Types of Annotations
Each type of annotation has a name, description, and color to use. There are three built-in categories which are shared by all Panorama folders on the server. You may change them or the colors they use, but be aware that other projects may be impacted by your changes. Annotation Types defined in the "Shared" project are available throughout the server. Types defined at the project level are available in all subfolders.
Instrumentation Change
Reagent Change
Technician Change
You can also define your own annotation types as required. For example, you might want to note changes to environment like addition of building HVAC or power outages. Our example above shows a "Candy Change" illustrating that you can decide what you think is worth noting.If you wish to Add new categories of annotations, use (Insert data) > Insert New Row in the QC Annotation Types section.To Edit existing annotation types, hover over the row and click the (Edit) icon that will appear in the leftmost column.
Add New Annotations to Plots
To enter a new annotation, use (Insert data) > Insert New Row in the QC Annotations section. Select the type of event (such as Reagent Change), enter a description to show in hover text ("new batch of reagent"), and enter when it occurred. Dates that include a time of day should be of the form "2013-8-21 7:00", but a simple date is sufficient. Return to the Panorama Dashboard tab to view the plots with your new annotation applied.The annotation symbol is placed on the x-axis above the tick mark for the date on which the event occurred. If there are multiple tickmarks for that date, the annotation will appear above the leftmost one. If an annotation occurred on a date for which there is no other data, a new tick mark will be added to the x-axis for that date.
Replicate Annotations
If the Skyline document included replicate annotations, you will see them listed in the ReplicateAnnotation panel at the bottom of the page. To learn about filtering QC plots using replicate annotation information, see this topic
Pareto plots combine a bar plot and a line plot, and are used to quickly identify which metrics are most indicative of a quality control problem. Each bar in the plot represents a metric (see metric code below). Metric bars are ordered by decreasing incidence of outliers, where outliers are defined as the number of instances where each metric falls outside of the +/- 3 standard deviation range. The line shows the cumulative outliers by percentage.There are separate pareto plots for each guide set and plot type (Levey-Jennings, Moving Range, CUSUMm, and CUSUMv) combination.Other items to note in Pareto plots:
Hover over dots in the line plot to show the cumulative %.
Hover over a metric bar to show the number of outliers.
Click a metric bar to see the relevant QC plot and guide set for that metric.
Print Plots
Hover over any plot to reveal PDF and PNG buttons. Click to export the Pareto plot for that guide set.
Indexed Retention Time (iRT) is a scheme for calibrating retention times across runs. For example, you might use an iRT scale like this set of 11 synthetic peptides from Biognosys. Based on these, peptides, the set of iRT anchor points can be extended to other peptides. The iRT of a peptide is a fixed number relative to a standard set of reference iRT-peptides that can be transferred across labs and chromatographic systems. iRT can be determined by any lab and shared transparently.This topic describes how you can use metrics related to the iRT calculations as system suitability metrics in Panorama. Monitoring the slope, intercept, and R-squared correlation values for the regression line can help mass spectrometrists confirm that equipment is performing as expected.
The iRT metrics (slope, intercept, and R-squared correlation) are scoped to each sample file. Panorama will detect whether to include these metrics alongside other metrics available in QC plots based on whether the Skyline document is configured to monitor iRT. Learn more about plot options in this topic: Panorama QC Plots
iRT Correlation
iRT Intercept
iRT Slope
Hover Details
As with other plotted metrics, hover over any point in the plot to see details including the value and acquired date.From the hover panel, you can also choose to exclude the sample from QC if desired. Learn more in this topic: Panorama QC Plots
Notifications
Users can subscribe to receive notifications when iRT metric values are out of range, as for other outlier values. Learn about outlier notifications in this topic: Panorama: Outlier Notifications.
An administrator can control which QC metrics are available for quality control plots on a folder-by-folder basis.Select the way in which you want metrics displayed in the current folder, and for the enabled metrics, how to identify outliers. Quieting display of outliers or disabling metrics may be practical for a variety of reasons, including if they produce many outliers without being diagnostic of real problems.
Levey-Jennings (+/- standard deviations): (Default)
Fixed value cutoff
Fixed deviation from mean
Show metric in plots, but don't identify outliers
Disabled, completely hide the metric: Never show the metric in this folder.
The configuration menu is available on the QC Plots web part.
In the upper right of the QC Plots web part, open the (triangle) pulldown menu.
Select Configure QC Metrics.
The Type column indicates whether a metric is scoped to a single peptide/molecule (precursor) or to a run as a whole.
Use the Metric usage pulldowns to select the desired usage for each metric in this folder.
Administrators can customize how outliers are defined per metric. These columns are only shown for Levey-Jennings, Fixed value cutoff, and Fixed deviation from mean selections in the Metric usage column.By default, +/- 3 standard deviations are used for upper and lower bounds with Levey-Jennings outlier identification. On the Configure QC Metrics page, edit the Lower bound/Upper bound columns.Fixed value cutoffs establish upper and lower bounds as discrete values, and are used for all precursors. It is useful for metrics like dot products (where one might set a lower bound of 0.90 and no upper bound), or mass error (where one might set a lower and lower bounds of -5 and 5 PPM.Fixed deviation from the mean are applied on a per precursor basis, using the mean of the metric for that precursor. It is useful for metrics like retention time, where one might set lower and upper bounds of -1 and 1 to identify retention time shifts of one minute in either direction.Means and standard deviations are calculated based on guide sets. If no guide sets are defined, all of the data for that metric is used.
Define Custom Metrics
An administrator can create their own custom metric by first defining a query, typically in the targetedms schema, with the following columns:
Name
Type
Description
Notes
MetricValue
NUMERIC
The value to be plotted
SampleFileId
INT
Value of the related Id in targetedms.SampleFile
PrecursorChromInfoId
INT
Value of the related Id in targetedms.PrecursorChromInfo. May be NULL, in which case there will be click handler for the point.
SeriesLabel
VARCHAR
The peptide sequence, custom ion name, etc.
DataType
VARCHAR
How to label the series in the overall list (typically either "Peptide", "Fragment", etc.)
PrecursorId
INT
Value of the related Id in the targetedms.Precursor or targetedms.MoleculePrecursor table.
mz
NUMERIC
mass to charge ratio
The fields PrecursorChromInfold must be included in a query for run-scoped metrics but should return NULL.
The fields SeriesLabel, DataType, PrecursorId, and mz are optional. They will be automatically calculated if they are missing, but you can use them to customize the display.Once your query is defined, reopen the Configure QC Metrics page from the web part menu and click Add New Custom Metric.In the popup:
Enter a Name
Select the Schema and Query for Series 1.
Give provide an Axis Label.
If you would like a second Y-axis tracking a different schema and query, specify those in Series 2.
Select the Metric Type:
Precursor: This metric varies for each individual molecule.
Run: This metric will have a single value per run, such as a maximum value of a pressure trace during a run.
To be able to enable your metric based on the presence of data, supply the following:
Enabled Schema: Schema in which to find the "Enabled" query.
Enabled Query: Query to determine when to enable the metric.
Click Save.
The new metric will appear on the metric configuration list as a clickable link. Click to see the definition. You can edit and save or delete if if necessary.When enabled, your new metric will also be available in the QC Plots pulldown menu for plotting. If you created a molecule-scoped metric, each series will be titled with the value of the "SeriesLabel" column from the backing query.
Example: Peptide ID Count
A real world example of adding a custom metric is to track the peptide ID counts in runs imported into the Skyline document in the folder. You can find an example of the steps used to create this metric, as well as an example data folder on PanoramaWeb:
Pressure traces, total ion current, and other chromatograms help monitor the performance of the mass spectrometry equipment. For example, a higher than normal value in system back pressure during the acquisition probably indicates the system is overpressure or starting to leak. Skyline will automatically extract pressure traces and other sample-scoped chromatograms from the raw files. When they're present, you can can monitor this at specific time points by configuring a metric to pull the pressure trace within the chromatogram at specific points in time, or track the time that a specified pressure value is reached.There are two options for these metrics:
Track trace value at user defined time (in minutes)
Track time of user defined trace value
Open the Configure QC Metrics page from the web part menu and click Add New Trace Metric. In the popup, specify:
Metric Name: You can use the name to give users details about the metric.
Use Trace: Select the trace to use. If this box reads "No trace can be found", you will not be able to create a trace metric.
Y Axis Label: You can use this label to give information about units (PSI, etc.) as needed.
Select the type of trace with the radio buttons:
Use trace value when time is greater than or equal to: Enter the number of minutes in the box.Use time when the trace first reaches a value greater than or equal to: Enter the desired value in the box.
In this topic, learn how any non-guest Panorama user can customize the notifications they receive. Opt-in to notifications when any data is uploaded, or only when the number of outliers in a specified number of files exceeds a custom threshold.
Subscribe to Notifications
Any logged-in user can configure their own subscription to notifications about imports and outliers. An unusual number of outliers or pattern sustained over multiple runs may indicate a problem with an instrument that should be corrected. Notifications are configured by folder - typically one folder monitors a single instrument.
To subscribe to notifications:
For the current folder/instrument, click Notifications in the top/main section of the QC Summary web part.
You can also select Subscribe to Outlier Notification Emails from the (triangle) menu.
For another folder/instrument, click Notifications in the appropriate panel (shown in the gray box below).
Use the radio buttons to control whether email notifications are sent to you. The default is Do not email with notifications about newly imported data.
If you select Email me when... you can also specify the threshold for triggering email:
Number of runs to consider, from 1 to 5. For one run, select most. For two runs, select two most, etc.
Number of outliers per file that will trigger the notification. Valid values from 0 to 999. To be notified of every upload regardless of presence of outliers, enter 0 (zero).
Click Save.
To unsubscribe, reopen the notification page, select Do not email with notifications about newly imported data, and click Save.
Notification Content
The notification will be sent immediately following a triggering upload and will have the email subject:
Panorama QC Outlier Notification - <##> outliers in file <filename> in <folder>
The body of the email will be the table shown in the QC Summary plot for a single sample file.
The Panorama QC folder is designed to help labs perform QC of their instruments and reagents over time. Runs are uploaded using the data pipeline or directly from Skyline. This topic describes how quality control guide sets can help track performance of metrics over time.
Guide sets give you control over which data points are used to establish the expected range of values in a QC plot. Instead of calculating the expected ranges based on all data points in the view, you can specify guide sets based on a subset of data points.You create a guide set by specifying two dates:
training start date
training end date
The training start and end dates (called the "training period") establish the period to calculate the expected range of values. The data points within the training period are used to calculate the mean and standard deviation for that guide set's expected range.Standard deviations are shown as colored bars: green for +/-1, blue for +/-2, and red for +/-3 standard deviations from the mean. The expected range calculated by a training period is applied to all future data points, until a new training period is started.The training periods are shown with a grey background. Hover over the training area to see detailed information about that guide set. You can also see these details on the Guide Sets tab.
Define Guide Sets
You can create guide sets directly from the QC plot. To add a new guide set, first uncheckGroup X-Axis Values by Date and Show All Series in a Single Plot. Next click Create Guide Set, drag to select an area directly on the graph, and click the Create button that appears over the selected area.Note: a warning will be given if fewer than 5 data points are selected, and you cannot create overlapping guide sets.Alternatively, you can create guide sets by entering the start and end dates manually: click the Guide Sets tab and click Insert New to manually enter a new guide set. Note that you must have been granted the Editor role or greater to create guide sets from either of these two methods.
Edit or Delete Guide Sets
To edit or delete guide sets, click the tab Guide Sets. To edit, hover over a row to reveal buttons for a guide set that has already been created. Click the (pencil) icon to edit. To delete, check the target guide set and click the (Delete) button.
Show Reference Guide Set
In any QC Plot display, use the Show Reference Guide Set checkbox to always include the guide set data in QC plots. This box is checked by default.When zooming in to view QC data for a specific date range, the guide set data currently being used to identify outliers will always be shown, even if it is outside the plot date range.The guide set is shaded in gray, and separated from the "selected" plot range by a vertical line.
Chromatograms present in the Skyline file can be viewed from within Panorama with several display customization options. Depending on the type of chromatogram you are viewing, some or all of the features described in this topic will be available.
The details pages for proteins, peptides, small molecules, etc., include a panel of Chromatograms.
Display Chart Settings
Open the Display Chart Settings panel, if it is not already expanded.
Set the Chart Width and Height.
Use checkboxes to Synchronize the X- and/or Y-axis if desired.
Use the Replicates dropdown to multi-select the specific replicates for which you want to see the chromatograms, making it easier to compare a smaller number of interest.
If the file also contains Annotations you can similarly multi-select the ones to display here.
Click Update to apply your settings.
Choose Split or Combined Chromatograms
When there is chromatogram data for both the precursors and the transitions, you can select how to display them in the chromatograms:
Split
Combined
In cases where the precursors' signal is much greater than the transitions', the latter will be almost an imperceptible bump at the bottom of the combined plot, so choosing the split option makes it clearer to view the signals.
Associating sample metadata with results data, including targeted mass spectrometry data, is made simple when you use both Panorama and LabKey Sample Manager on the same server.Users can perform deeper analyses and quickly gain insight into which samples have data associated. Additionally, users can export sample data to help configure their instrument acquisition and Skyline documents.
Set up a LabKey Sample Manager folder on the server. To show Skyline/Panorama data, enable the targetedms module in that folder. Other Panorama projects and folders on the same server will be able to connect to this shared sample repository.Note that the terms "Sample Name" and "Sample ID" are interchangeable in this topic, and this walkthrough assumes that you already have Panorama data and want to link it to additional Sample metadata.Switch between the Sample Manager interface and Panorama (LabKey Server) interface using the product selection menu.
Link Panorama Data to Sample Metadata
To link samples in Panorama to metadata about them, first create the Sample Type with desired fields in Sample Manager. Your Sample Type definition can include as much information as you have or will need. For example, you might create a "Panorama Samples" type that included a project name and collection date, shown in the examples on this page.In any Panorama folder on the server, load your Skyline document(s) as usual. Click the Runs tab, click the name of the Skyline file, then click on the replicates link to get a list of Sample Names.Return to Sample Manager, and create new Samples for each of the Sample Names you want to link, populating the metadata fields you defined. You can do this in bulk or by file import, but for this small example, we add the metadata in a grid.Note that the sample metadata and Skyline documents can be imported in any order.Once you've registered the samples, return to the replicate list in Panorama and you will now see the sample metadata for your samples.Click the name of a linked sample here and you will return to the Sample Manager interface for that sample, where you could find it in storage, review assay data, etc.
Link from Samples to Panorama Data
When viewing the details for a sample in Sample Manager, if there is linked data in Panorama, you will see a tab for Skyline Documents on the Assays tab.If there were additional assay data about these samples, it would be shown on other tabs here. Notice you can also track lineage, aliquots, workflow jobs, and the timeline for any sample linked in this way.Click the name of the File to go to the Panorama data for that sample, which could be in another project or folder on your server.
Provide Sample List to Acquisition Software
Users may want to export a list of samples to their acquisition software. This will reduce errors and improve efficiency.Within Sample Manager:
Open the Panorama sample type you created and identify the desired samples by filtering, sorting, or creating a picklist.
Select the desired samples to include in your acquisition list.
Export the grid of samples to Excel, csv, or tsv file.
If the default set of exported fields does not match your requirements, you could edit it offline to remove extraneous columns.A user could also create a custom grid view of samples specifically tailored to the requirements. When present, such a grid could be selected from the Grid Views menu:
Collaboration within teams and between organizations is key to modern integrated research. LabKey Server is specifically designed to make working together easy and secure.
Basic tools like message boards, issue trackers, and wiki documents let users communicate on highly customized and centralized web pages.
Role and group based security ensures that the right people can see and change information, and makes access easy to administer.
Whatever your organization's workflow, you can design tools within LabKey to improve collaboration.
Collaboration Tools
File management -- Share and manage files using a secure repository
Message boards -- Facilitate team discussions about your project.
Issue tracking -- Solve multi-step problems requiring input from many team members.
Wiki pages -- Document and contextualize your project.
Adjudication Module -- Tools for increased diagnosis confidence through independent determinations.
Contacts -- When enabled, create a user contacts page.
Get Started with Collaboration Tools
To learn to use the collaboration tools, use the following topics. You do not need to complete them all or follow them in this order.
These tutorial topics can all be completed using a free trial version of LabKey Server. It takes a few minutes to set up and you can use it for 30 days.
Tutorial Topics
Learn with a LabKey Server Trial
The easiest way to learn to use the tools is with a trial version of LabKey Server. You can explore tools that are already created for you in the pre-loaded example project.
Exploring LabKey Collaboration: Complete the "Tour" and "Try It Now" sections to learn to use message boards, issue trackers, and wikis.
Use a File Browser: The "Laboratory Data" example folder includes a file browser.
LabKey Server provides both a file repository and a database for securely storing, sharing and integrating your information.
Browser-based, secure upload of files to LabKey Server where you can archive, search, store, and share.
Structured import of data into the LabKey database, either from files already uploaded or from files stored outside the server. The imported data can be analyzed and integrated with other data.
Browser-based, secure sharing and viewing of files and data on your LabKey Server.
Basic Functions
Once files have been uploaded to the repository, they can be securelysearched, shared and viewed, or downloaded. Learn the basics in this tutorial:
Once data has been imported into the database, team members can integrate it across source files, analyze it in grid views (using sort, filter, charting, export and other features), perform quality control, or use domain specific tools (eg., NAb, Flow, etc.). The basic functions described above (search, share, download and view) remain available for both the data and its source files.
Application Examples
Members of a research team might upload experimental data files into the repository during a sequence of experiments. Other team members could then use these files to identify a consistent assay design and import the data into the database later in a consistent manner. Relevant tutorials:
LabKey Server makes it easy to centralize your files in a secure and accessible file repository. This tutorial shows you how to set up and use some of the LabKey Server file management tools.
Researchers and scientists often have to manage large numbers of files with a wide range of sizes and formats. Some of these files are relatively small, such as spreadsheets containing a few lines of data; others are huge, such as large binary files. Some have a generic format, such as tab-separated data tables; while others have instrument-specific, proprietary formats, such as Luminex assay files -- not to mention image-based data files, PowerPoint presentations, grant proposals, longitudinal study protocols, and so on.Often these files are scattered across many different computers in a research team, making them difficult to locate, search over, and consolidate for analysis. Worse, researchers often share these files via email, which puts your data security at risk and can lead to further duplication and confusion.
Solution: LabKey Server File Repository
LabKey Server addresses these problems with a secure, web-accessible file repository, which serves both as a searchable storage place for files, and as a launching point for importing data into the database (for integration with other data, querying, and analysis).In particular, the file repository provides:
A storage, indexing and sharing location for unstructured data files like Word documents. The search indexer scans and pulls out words and phrases to enable finding files with specific content.
A launching point for structured data files (like Excel files or instrument-generated files with a specific format), that can be imported into the LabKey Server database for more advanced analysis. For example, you can select files in the repository, and import them into assay designs or process the files through a script pipeline.
A staging point for files that are opaque to the search indexer, such as biopsy image files.
This tutorial shows you how to set up and use a LabKey Server file repository that handles all three of these file types.
To begin, we'll set up the file repository user interface. The file repository has a web-based interface that users can securely interact with online, through a web browser. Only those users you have explicitly authorized will have access to the file repository. After the user interface is in place, we upload our data-bearing files to the repository.
Google Chrome is the recommended browser for this step.
Set up a File Repository
First we will set up the workspace for your file repository, which lets you upload, browse, and interact with files.
Log in to your server and navigate to your "Tutorials" project. Create it if necessary.
If you don't already have a server to work on where you can create projects, start here.
If you don't know how to create projects and folders, review this topic.
Create a new subfolder named "File Repository". Accept all defaults.
Enter > Page Admin Mode.
Using the (triangle) menus in the corner of each web part, you can Remove from page the Subfolders, Wiki, and Messages web parts; they are included in a default "Collaboration" folder but not used in this tutorial.
Drag and drop the unzipped folder LabKeyDemoFiles, onto the target area on the File Repository.
Notice the progress bar displays the status of the import.
Click the (Toggle Folder Tree) button on the far left to show the folder tree.
When uploaded, the LabKeyDemoFiles folder should appear at the root of the file directory, directly under the fileset node.
Securing and Sharing the Repository (Optional)
Now you have a secure, shareable file repository. Setting up security for the repository is beyond the scope of this tutorial. To get a sense of how it works, go to > Folder > Permissions. The Permissions page lets you grant different levels of access, such as Reader, Editor, Submitter, etc., to specified users or groups of users. Uncheck the "Inherit permissions from parent" box if you wish to make changes now. For details on configuring security, see Tutorial: Security. Click Cancel to return to the main folder page.
In the previous tutorial step, you created and populated a file repository. Users of the file repository can browse and download files, but administrators of the repository have an expanded role. When you created your own folder for this tutorial, you automatically received the administrator role.As an administrator, you can:
Add and delete files.
Customize which actions are exposed in the user interface.
Audit user activity, such as when users have logged in and where they have been inside the repository.
We will begin by interacting with the repository as an ordinary user would, then we will approach the repository as an administrator with expanded permissions.
Click the toggle to show the folder tree on the left, if it is not already visible.
Note that if there are more buttons visible than fit across the panel, the button bar may overflow to a >> pulldown menu on the right.
Click into the subfolders and files to see what sort of files are in the repository.
Double-click an item to download it (depending on your browser settings, some types of files may open directly).
Customize the Button Bar (Admins)
You can add, remove, and rearrange the buttons in the Files web part toolbar. Both text and icons are optional for each button shown.
Return to the File Repository folder if you navigated away.
In the Files web part toolbar, click Admin.
If you don't see the admin button, look for it on a >> pulldown menu on the right. This overflow menu is used when there are more buttons shown than can fit across the window.
Select the Toolbar and Grid Settings tab.
The Configure Toolbar Options shows current toolbar settings; you can select whether and how the available buttons (listed on the right) will be displayed.
Uncheck the Shown box for the Rename button. Notice that unsaved changes are marked with red corner indicators.
You can also drag and drop to reorder the buttons. In this screenshot, the parent folder button is being moved to the right of the refresh button.
Make a few changes and click Submit.
You may want to undo your changes before continuing, but it is not required as long as you can still see the necessary buttons. To return to the original file repository button bar:
Click Admin in the Files web part toolbar.
Click the Toolbar and Grid Settings tab.
Click Reset to Default.
Configure Grid Column Settings (Admins)
Grid columns may be hidden and reorganized using the pulldown menu on the right edge of any column header, or you can use the toolbar Admin interface. This interface offers control over whether columns can be sorted as well.
In the Files web part toolbar, click Admin.
On the Toolbar and Grid Settings tab, scroll down to Configure Grid Column Settings.
Using the checkboxes, select whether you want each column to be hidden and/or sortable.
Reorder columns by dragging and dropping.
Click Submit when finished.
If needed, you may also Reset to Default.
Audit History (Admins)
The Audit History report tells an administrator when each file was created or deleted and who executed the action.
In the Files web part, click Audit History.
In this case you will see when you uploaded the demo files.
Files in the repository, both structured and unstructured, are indexed using the full-text search scanner. This is different from, and complimentary to, the search functionality provided by SQL queries, covered in this topic: SearchIn this tutorial step you will search your files using full-text search and you will add tags to files to support more advanced search options.
Add Search User Interface
Return to the File Repository folder where you installed the Files web part.
The search results show a variety of documents that contain the search term "serum".
Click the links to view the contents of these documents.
Try other search terms and explore the results. For example, you would see an empty result for "Darwin" or "Mendel" before you complete the next part of this tutorial.
Click Advanced Options for more options including:
specifying the desired project and folder scope of your search
narrowing your search to selected categories of data
File Tagging
In many cases it is helpful to tag your files with custom properties to aid in searching, for example, when the desired search text is not already part of the file itself. For example, you might want to tag files in your repository under their appropriate team code names, say "Darwin" and "Mendel", and later retrieve files tagged for that team.To tag files with custom properties, follow these steps:
Define a 'Team' Property
Click the File Repository link.
In the Files web part header, click Admin.
If you don't see it, try the >> pulldown menu or check your permissions.
In the Name field, enter "Team". Leave the Text type selected.
Click Save.
Learn more about adding and customizing fields in this topic: Field Editor.
Apply the Property to Files
Open the folder tree toggle and expand directories.
Select any two files using their checkboxes.
Click (Edit Properties). It may be shown only as a wrench-icon. (Hover over any icon-only buttons to see a tooltip with the text. You might need to use the files web part Admin > Toolbar and Grid Settings interface to make it visible.)
In the Team field you just defined, enter "Darwin" for the first file.
Click Next.
In the Team field, enter "Mendel".
Click Save.
Retrieve Tagged Files
In the Search web part, enter "Darwin" and click Search. Then try "Mendel".
Your tagged files will be retrieved, along with any others on your server that contain these strings.
Turn Search Off (Optional)
The full-text search feature can search content in all folders where the user has read permissions. There may be cases when you want to disable global searching for some content which is otherwise readable. For example, you might disable searching of a folder containing archived versions of documents so that only the more recent versions appear in project-wide search results.
To turn the search function off in a given folder, first navigate to it.
Select > Folder > Management, and click the Search tab.
Remove the checkmark and click Save.
Note that you can still search from within the folder itself for content there. This feature turns off global searches from other places.
The file repository serves as a launching point for importing assay data to the database. First, notice the difference between uploading and importing data:
You already uploaded some data files earlier in the tutorial, adding them to the file system.
When you import a file into LabKey Server, you add its contents to the database. This makes the data available to a wide variety of analysis/integration tools and dashboard environments inside LabKey Server.
In this last step in the tutorial, we will import some data of interest and visualize it using an assay analysis dashboard.
Import Data for Visualization
First, import some file you want to visualize. You can pick a file of interest to you, or follow this walkthrough using the sample file provided.
Return to the main page of the File Repository folder.
In the Files web part, locate or upload the data file. To use our example, open the folders LabKeyDemoFiles > Assays > Generic.
Select the file GenericAssay_Run1.xls.
Click Import Data.
In the Import Text or Excel Assay, select Create New General Assay Design.
Click Import.
Enter a Name, for example, "Preliminary Lab Data".
Select "Current Folder (File Repository)" as the Location.
Notice the import preview area below. Let's assume the inferred columns are correct and accept all defaults.
Click Begin Import, then Next to accept the default batch properties, then Save and Finish.
When the import completes, you see the "list" of runs, consisting of the one file we just imported.
Click View Results to see the detailed view of the assay data.
LabKey Server makes it easy to upload and use your files in a secure and accessible file repository. Users work with files in a Files web part, also known as a file browser. This topic covers setting up and populating the file browser. You can also learn to use files in the Tutorial: File Repository.
An administrator can add a file browser to any folder or tab location.
Navigate to the desired location.
Enter > Page Admin Mode.
In the lower left, select Files from Add Web Part and click Add.
Click Exit Admin Mode.
There is also a file browser available in the narrow/righthand column of webparts. It offers the same drag and drop panel, but the menu bar is abbreviated, concealing the majority of menu options behind a Manage button.
Drag and Drop Upload
The Files web part provides a built-in drag-and-drop upload interface. Open a browser window and drag the desired file or files onto the drag-and-drop target area.Folders, along with any sub-folders, can also be uploaded via drag-and-drop. Uploaded folder structure will be reproduced within the LabKey file system. Note that empty folders are ignored and not uploaded or created.While files are uploading, a countdown of files remaining is displayed in the uploader. This message disappears on completion.
Single File Upload Button
If drag and drop is not suitable, you can also click the Upload Files button which will let you browse to Choose a File. You can also enter a Description via this method. Click Upload to complete.
Create Folder Structure in the File Repository
When you import a files in a folder/subfolder structure, that structure is retained, but you can also create new folders directly in the file browser. In the file browser, select the parent folder, or select no parent to create the new folder at the top level. Click the Create Folder button:Enter the name of the new folder in the popup dialog, and click Submit:Note that folder names must follow these naming rules:
The name must not start with an 'at' character: @
The name must not contain any of the following characters: / \ ; : ? < > * | " ^
If you would like to create multiple distinct folder structures within a single folder, make use of named file sets.
To show the column containing the file paths to users granted the necessary permission:
In the file browser, click Admin.
Click the Toolbar and Grid Settings tab.
Scroll down to the Configure Grid Column Settings section.
Uncheck the box for Hidden next to Absolute File Path (permission required).
Users who have not been given the See Absolute File Paths role will see only an empty column.Whether the absolute path has been exposed in the files browser or not, clicking into the Manage Files page (by clicking the web part title) will also show authorized users the absolute path along the bottom of the window.
This topic describes how users can view and share files that have been uploaded to the files repository. Collaborating with a shared file browser makes it easier to ensure the entire team has access to the same information.
Learn more about creating a file repository and uploading files in this topic: Using the Files Repository.Learn about using the data processing pipeline to upload on a schedule, in bulk, or with special processing in this topic: Data Processing Pipeline. Pipeline tasks can also be run on files already uploaded to the repository as they are imported into the database providing further processing and analysis.
File Preview
When you hover over the icon for some types of files in the Files web part, you'll see a pop-up showing a preview of the contents for a few seconds. This is useful for quickly differentiating between files with similar names, such as in this image showing numbered image files.
Select Files
To select a single file, use the checkbox on the left. You can also click the file name to select that file.Many actions will operate on multiple files. You can check multiple boxes, or multiselect a group of files by selecting the first one, then shift-clicking the name of the last one in the series you want to select.
File Display in Browser
Double-clicking will open some files (i.e. images, text files including some scripts, etc) directly, depending on browser settings. Other files may immediately download.
File Download Link
For flexible sharing of any file you have uploaded to LabKey Server, make a download link column visible.
Navigate to the Files web part showing the file.
Make the Download Link column visible using the (triangle) menu for any current column, as shown here:
Right click the link for the file of interest and select Copy link address.
The URL is now saved to your clipboard and might look something like:
This URL can be shared to provide (authorized) users with a link to download the file of interest.
Users will need to have sufficient permissions to see the file. For information on adjusting permissions, please see Security.For file types that can be displayed in the browser, you can edit the download to display the content in the browser by removing this portion of the URL:
Note that this simple method for showing the Download Link column is not permanent. To configure your files web part to always display this column, use the file browser Admin menu, Toolbar and Grid Settings tab instead.
An alternative way to get the download link for a single file:
Click the title of the Files web part to open the manage page.
On the Manage Files page, place a checkmark next to the target file.
At the bottom of the page, copy the WebDav URL.
Get the Base URL
To get the base URL for the File Repository, select the fileset item on the Manage Files page, or from anywhere in the folder, go to > Go To Module > FileContent. The base URL is displayed at the bottom of the File Repository window, as shown below:
Link to a Specific Directory in the File Repository
To get the link for a specific directory in the repository, navigate to that directory and copy the value from the WebDAV URL field. Notice the subdirectories are appended to the base folder URL. For example, the following URL points to the directory "LabKeyDemoFiles/Assays/Generic" in the repository.
Use 'wget' to Transfer Files from the Command Line
Use command line methods such as 'wget' to access file system files, given the path location. For example, if there is a file in your File Share browser called "dataset.xls", you (as an authorized user) could download that file via the wget command like this, substituting the path to your @files folder:
This would prompt you to provide your password for labkey.org and the file called dataset.xls will download to your computer in whatever folder you are are in at the time you ran the command.If you have a file to upload to your file browser, you can also use wget to simplify this process. In this example, user "Smith" wants to deliver a file named "dataset.xls" to the "ClientProject" support portal. They could run the following command, remembering to correct the path names for the client portal and file share folder:
After answering the password prompt and logging into labkey.org, the file would go from the designated location straight to the file repository on labkey.org. Remember that users must already be authorized to add files to the repository for this command to work.
Files in a LabKey file repository can be displayed, or rendered, in different ways by editing the URL directly. This grants you flexibility in how you can use the information you have stored.This example begins with an HTML file in a file repository. For example, this page is available at the following URL:
By default, this file will be rendered inside the standard server framing:To render the content in another way, add the renderAs parameter to the URL. For example to display the file without any framing, use the following URL
Render the content of the file directly into a page. This is only useful if you have files containing fragments of HTML, and those files link to other resources on the LabKey Server, and links within the HTML will also need the renderAs=INLINE to maintain the look.
If the target files are in a named file set, you must add the fileSet parameter to the URL. For example, if you are targeting the file set named "store1", then use a URL like the following:
Once you have uploaded files to LabKey Server's file repository, you can import the data held in these files into LabKey's database via the Files web part. After import, data can be used with a wide variety of analysis and visualization tools.
Import Data from Files
Before you can import data files into LabKey data structures, you must first upload your files to the LabKey file system using the Files web part, pipeline, or other method.After you have uploaded data files, select a file of interest and click the Import Data button in the Files web part.In the Import Data pop up dialog, select from available options for the type of data you are importing.Click Import to confirm the import.Some data import options will continue with additional pages requesting more input or parameters.
The following topic explains how to automatically link together assay data with images and other file types in the File repository.
When imported assay data contains file names and/or paths, the system can associate these file names with actual files residing on the server. Files are resolved for both assay results and assay run properties. Assay results file can be either TSV or Excel formats.For Standard (previously known as "GPAT") assays, files will be automatically associated with assay result runs provided that:
The files have been uploaded to a location on the LabKey Server file repository.
The assay design is of type Standard
The file names and paths are captured in a field of type File.
The file names resolve to a single file
Paths are either full or relative paths to the pipeline or file repository root.
File names will be associated with the actual images in the following scenarios:
On import:
When assay data is imported from the file browser.
When assay data is uploaded via the assay upload wizard (in this case, we use the pipeline root).
When an assay run is created via experiment-saveBatch.api.
When an assay run is created via assay-importRun.api.
On update:
When updating assay data via query-updateRows.api.
When updating assay data via experiment-saveBatch.api.
When updating assay data via the build-in browser-based update form.
Note that image files are exported with assay results as follows:
when exported as Excel files, images appear inline.
when exported as .tsv files, the file name is shown.
Example
Assume that the following image files exist in the server's file repository at the path scans/patient1/CT/When the following assay result file is imported...
ParticipantId
Date
ScanFile
100
12/12/2017
scans/patient1/CT/ct_scan1.png
100
12/12/2017
scans/patient1/CT/ct_scan2.png
100
12/12/2017
scans/patient1/CT/ct_scan3.png
...the imported assay results will resolve and link to the files in the repository. Note the automatically generated thumbnail in the assay results:
API Example
File association also works when importing using the assay API, for example:
Suppose you have a dataset where each of row of data refers to some image or file. For example, you have a dataset called Biopsies, where you want each row of data to link to an image depicting a tissue section. Images will open in a new tab or download, depending on your browser settings.Below is an example Biopsies table:
Biopsies
ParticipantId
Date
TissueType
TissueSlide
PT-101
10/10/2010
Liver
slide1.jpg
PT-102
10/10/2010
Liver
slide2.jpg
PT-103
10/10/2010
Liver
slide3.jpg
How do you make this dataset link to the slide images, such that clicking on slide1.jpg shows the actual image file?
Solution
To achieve this linking behavior, follow these steps:
Drag-and-drop your files into the File Repository. You can upload the images directly into the root directory, or you can upload the images inside a subfolder. For example, the screenshot below, shows a folder called images, which contains all of the slide JPEGs.
Acquire the URL to your image folder: In the File Repository, open the folder where your images reside, and scroll down to the WebDav URL.
Open a text editor, and paste in the URL, for example:
Your dataset should include a column which holds the file names of your target images. See "Biopsies" above for an example.For details on importing a study dataset, see Import Datasets.To edit an existing dataset to add a new column, follow the steps in this topic: Dataset Properties.
Build the URL
To build the URL to the images, do the following:
In your dataset, determine which column holds the image names. In our example the column is "TissueSlide".
In a text editor, type out this column name as a "substitution token", by placing it in curly brackets preceded by a dollar sign, as follows:
${TissueSlide}
Append this substitution token to the end of the WebDav URL in your text editor, for example:
You now have a URL that can link to any of the images in the File Repository.
Use the URL
Go the Dataset grid view and click Manage.
Click Edit Definition.
Click the Fields section.
Expand the field which holds the image names, in this example, "TissueSlide".
Into the URL field, paste the full URL you just created, including the substitution token.
Click Save.
Click View Data.
Notice that the filenames in the TissueSlide field are now links. Click a link to see the corresponding image file. It will open in a new tab or download, depending on your browser.
If you prefer that the link results in a file download, add the following to the end of the URL in the dataset definition.
?contentDisposition=attachment
Resulting in this total URL on the Display tab for the TissueSlide field.
When files are added to the File Repository, metadata about each file is recorded in the table exp.Data.
The information about each file includes:
File Name
URL
Download Link
Thumbnail Link
Inline Thumbnail image
etc...
View the Query Browser
To access the table, developers and admins can:
Go to > Go to Module > Query.
Click to open the schema exp and the query Data.
Click View Data.
By default, the name, runID (if applicable) and URL are included. You can use the (Grid views) > Customize Grid option to show additional columns or create a custom grid view.
Create a Query Web Part
To create a grid that shows the metadata columns, do one of the following:
Create a custom query using exp.Data as the base query, and add the columns using SQL. For example, the following SQL query adds the columns InlineThumbnail and DownloadLink to the base query:
SELECT Data.Name, Data.Run, Data.DataFileUrl, Data.InlineThumbnail, Data.DownloadLink FROM Data
Pulling Metadata from File Paths
You can pull out more metadata, if you conform to a path naming pattern.
In the following example, CT and PET scans have been arranged in the following folder pattern:
The following SQL query harvests out the patient id and the scan type from the directory structure above.
SELECT Data.Name as ScanName, Data.DataFileUrl, split_part(DataFileUrl, '/', 8) as Ptid, split_part(DataFileUrl, '/', 9) as ScanType FROM Data WHERE locate('/@files/scans', DataFileUrl) > 0
The result is a table of enhanced metadata about the files, which provides new columns you can filter on, to narrow down to the files of interest.
Administrators can customize the Files web part in the following ways, allowing them to present the most useful set of features to their file browser users.
To customize the Files web part, click the Admin button on the toolbar.There are four tabs for the various customizations possible here:
Customize Actions Available
The Actions tab lets you control the availability of common file action and data import options such as importing datasets, creating assay designs, etc.
Note that file actions are only available for files in the pipeline directory. If a pipeline override has been set, or the file browser is showing non-file content (such as in a "@fileset" directory), these actions will not be available in the default file location.You can change what a Files web part displays using the (triangle) menu; choose Customize. Select a file content directory to enable actions on this tab.
The Import Data button is always visible to admins, but you can choose whether to show it to non-admin users. For each other action listed, checking the Enabled box makes it available as a pipeline job. Checking Show in Toolbar adds a button to the Files web part toolbar.
Define File Tagging Properties
The File Properties tab lets you define properties that can be used to tag files. Each file browser can use the default (none), inherit properties from the parent container, or use custom file properties.Once a custom property is defined, users are asked to provided property values when files are uploaded. There is a built in 'Comment' file property that is included when other custom properties are in use.
To define a property, select Use Custom File Properties and then click the Edit Properties button.
Click Add Field to add new file properties. Details about customizing fields are available in this topic: Field Editor.
To reuse properties defined in the parent folder, select Use Same Settings as Parent.
Tagged files can be retrieved by searching on their property value. For more detail, see Step 3: Search the Repository.
Control Toolbar Buttons and Grid Settings
The Toolbar and Grid Settings tab controls the appearance of the file management browser.Configure Toolbar Options: Toolbar buttons are in display order, from top to bottom; drag and drop to rearrange. Available buttons which are not currently displayed are listed at the end. Check boxes to show and hide text and icons independently.Configure Grid Columns Settings (scroll down): lists grid columns in display order from top to bottom; drag and drop to rearrange. The columns can be reorganized by clicking and dragging their respective rows, and checkboxes can make columns Hidden or Sortable.You can also change which columns are displayed directly in the Files web part. Pulldown the arrow in any column label, select Columns and use checkboxes to show and hide columns. For example, this screenshot shows adding the Download Link column:Local Path to FilesUse the following procedure to find the local path to files that have been uploaded via the Files web part.
Go to > Site > Admin Console.
Under Configuration, click Files.
Under Summary View for File Directories, locate and open the folder where your Files web part is located.
The local path to your files appears in the Directory column. It should end in @files.
General Settings Tab
Use the checkbox to control whether to show the file upload panel by default. The file upload panel offers a single file browse and upload interface and can be opened with the Upload Files button regardless of whether it is shown by default.You may drag and drop files into the file browser region to upload them whether or not the panel is shown.Note that in Premium Editions of LabKey Server, file upload can be completely disabled if desired. See below for details.
Configure Default Email Alerts
To control the default email notification for the folder where the Files web part resides, see Manage Email Notifications. Email notifications can be sent for file uploads, deletions, changes to metadata properties, etc. Admins can configure each project user to override or accept the folder's default notification setting at the folder level.Project users can also override default notification settings themselves by clicking the (Email Preferences) button in the Files web part toolbar. SElect your own preference and click Submit.
Disable File Upload
Premium Feature — This feature is available in Premium Editions of LabKey Server. Learn more or contact LabKey.
Site Administrators can disable file upload across the entire site to both the file and pipeline roots. Disabling upload prevents use of the Upload button as well as dragging and dropping files to trigger file upload. Users with sufficient permissions can still download, rename, delete, move, edit properties, and create folders in the file repository.Note that attaching files to issues, wikis, or messages is not disabled by this setting.Learn more in this topic:
Premium Resource AvailableSubscribers to premium editions of LabKey Server can configure scanning of uploaded files for viruses. Learn more in this topic:
LabKey Server provides tools for securely uploading, processing and sharing your files. If you wish, you can override default storage locations for each project and associated subfolders by setting site-level or project-level file roots.
Note that if you are using a pipeline override, it will supersede settings described on this page for file roots.
Summary View of File Roots and Overrides
You can view an overview of settings and full paths from the "Summary View for File Directories" section of the "Configure File System Access" page that is available through > Site > Admin Console > Configuration > Files.File directories, named file sets and pipeline directories can be viewed on a project/folder basis through the "Summary View." The 'Default' column indicates whether the directory is derived from the site-level file root or has been overridden. To view or manage files in a directory, double click on a row or click on the 'Browse Selected' button. To configure an @file or an @pipeline directory, select the directory and click on the 'Configure Selected' button in the toolbar.
If you add a pipeline override for any folder, we don't create a @files folder in the filesystem. The server treats an override as a user-managed location and will use the path specified instead.Note that a @pipeline marker is used in the "Summary View for File Directories", available through > Site > Admin Console > Configuration > Files. However, there is no corresponding @pipeline directory on the file system. The summary view uses the @pipeline marker simply to show the path for the associated pipeline.
Site-Level File Root
The site-level file root is the top of the directory structure for files you upload. By default it is under the LabKey Server installation directory, but you may choose to place it elsewhere if required for backup, permissions, or disk space reasons.During server setup, a directory structure is created mirroring the structure of your LabKey Server projects and folders. Each project or folder is a directory containing a "@files" subdirectory. Unless the site-level root has been overridden at the project or folder level, files will be stored under the site-level root.You can specify a site-level file root at installation or access the "Configure File System Access" page on an existing installation.
Select > Site > Admin Console.
Under Configuration, click Files.
Change the Site-Level File Root
When you change the site-level file root for an existing installation, files in projects that use file roots based on that site-level file root will be automatically moved to the new location. The server will also update paths in the database for all of the core tables. If you are storing file paths in tables managed by custom modules, the custom module will need register an instance of org.labkey.api.files.FileListener with org.labkey.api.files.FileContentService.addFileListener(), and fix up the paths stored in the database within its fileMoved() method.Files located in projects that use pipeline overrides or in folders with their own project- or folder-level file roots will not be moved by changing the site-level file root. If you have set project-level roots or pipeline overrides, files in these projects and their subfolders must be moved separately. Please see Troubleshoot Pipeline and Files for more information.Changes to file roots are audited under Project and Folder events.
Block Potentially Malicious Files
Files with names containing combinations like " -" (a space followed by a hyphen or dash) can potentially be used maliciously (to simulate a command line argument). Administrators can block the upload of such files with two site-wide options.
Block file upload with potentially malicious names: Blocks direct upload and drag-and-drop of files.
Block server-side file creation with potentially malicious names: Also blocks server-side actions like import of folder archives containing wiki attachments with potentially malicious names, as these attachments are represented as "files" in the folder archive format.
Project-level File Roots
You can override the site-level root on a project-by-project basis. A few reasons you might wish to do so:
Separate file storage for a project from your LabKey Server. You might wish to enable more frequent backup of files for a particular project.
Hide files. You can hide files previously uploaded to a project or its subfolders by selecting the "Disable File Sharing" option for the project.
Provide a window into an external drive. You can set a project-level root to a location on a drive external to your LabKey Server.
The default file root for a folder is a subfolder of the project file root plus the folder name. If the project-level root changes, this folder-level default will also change automatically to be under the new project-level root.To set a custom file root for a single folder, follow these steps:
When you change the site-level file root for an existing installation, the entire directory tree of files located under that site-level file root are automatically moved to the new location (and deleted in the previous location).When you select a new project-level (or folder-level) file root, you will see the option "Proposed File Root change from '<prior option>'." Select what you want to happen to any existing files in the previous root location. The entire directory tree, including any subfolders within the file root are included in any copy or move.Options are:
Not copied or moved: Default
Copied to the new location: The existing files stay where they are. A copy of the files is placed in the new file root and database paths are updated to point to the new location.
Moved to the new location: Files are copied to the new location, database paths are updated to point to this location, then the original files are deleted.
Note: All work to copy or move the entire directory tree of files is performed in a single pipeline job. The job will continue even if some files fail to copy, but if any file copy fails, no deletion will occur from the original location (i.e. a move will not be completed).
File Root Sizes
The size of file roots is calculated during system maintenance and reported on the Files tab of the project settings and folder management pages.
Premium Feature AvailableSubscribers to the Professional and Enterprise Editions of LabKey Server can generate a site- or project-and-subfolder wide report of many file roots at once. Learn more here:
The directory exposed by the Files web part can be set to any of the following directories:
@files (the default)
@pipeline
@filesets
@cloud
any children of the above
Administrators can select which directory is exposed by clicking the (triangle) on the Files web part and selecting Customize. In the File Root pane, select the directory to be exposed and click Submit. The image below shows how to expose the sub-directory Folder A.
Alternative _webfiles Root
Administrators can enable an alternative WebDAV root for the whole server. This alternative webdav root, named "_webfiles", displays a simplified, file-sharing oriented tree that omits non-file content (like wikis), and collapses @file nodes into the root of the container’s node.To access or mount this root go to a URL like the following (replacing my.labkeyserver.com with your real server domains):
http://my.labkeyserver.com/labkey/_webfiles
This URL will expose the server's built-in WebDAV UI. 3rd party WebDAV clients can mount the above URL just like they can mount the default _webdav root.To enable this alternative webdav root:
Select > Site > Admin Console.
Under Configuration, click Files.
Under Alternative Webfiles Root, place a checkmark next to Enable _webfiles.
Click Save.
The _webfiles directory is parallel to the default _webdav directory, but only lists the contents under @files and its child containers. @pipeline, @filesets, and @cloud contents are not accessible from _webfiles.Any name collisions between between containers and file system directories will be handled as follows:Child containers that share names (case-insensitive) with file system directories will take precedence and be displayed with their names unchanged in the WebDAV tree. File system directories will be exposed with a " (files)" suffix. If there are further conflicts, we will append "2", "3", etc, until we find an unused name. Creating a subdirectory via WebDav always create a child file directory, never a child container.
Map Network Drive (Windows Only)
LabKey Server runs on a Windows server as an operating system service, which Windows treats as a separate user account. The user account that represents the service may not automatically have permissions to access a network share that the logged-in user does have access to. If you are running on Windows and using LabKey Server to access files on a remote server, for example via the LabKey Server pipeline, you'll need to configure the server to map the network drive for the service's user account.Configuring the network drive settings is optional; you only need to do it if you are running Windows and using a shared network drive to store files that LabKey Server will access.Before you configure the server below, your network directory should already be mapped to a letter drive on Windows.
Select > Site > Admin Console.
Under Configuration, click Files.
Under Map Network Drive, click Configure.
Drive letter: The drive letter to which you want to assign the network drive.Path: The path to the remote server to be mapped using a UNC path -- for example, a value like "\\remoteserver\labkeyshare".User: Provide a valid user name for logging onto the share; you can specify the value "none" if no user name or password is required.Password: Provide the password for the user name; you can specify the value "none" if no user name or password is required.
Named File Sets
Named file sets are additional file stores for a LabKey web folder. They exist alongside the default file root for a web folder, enabling web sharing of files in directories that do not correspond exactly to LabKey containers. You can add multiple named file sets for a given LabKey web folder, displaying each in its own web part. The server considers named file sets as "non-managed" file systems, so moving either the site or the folder file root does not have any effect on named file sets. File sets are a single directory and do not include any subdirectories.To add a named file root:
On the Files web part, click the (triangle) and select Customize.
On the Customize Files page, click Configure File Roots.
Under File Sets, enter a Name and a Path to the file directory on your local machine.
Click Add File Set.
Add additional file sets are required.
To display a named file set in the Files web part, click the (triangle) on the Files web part, and select Customize.
Open the "@filesets" node, select your named file set, and click Submit.
The Files web part will now display the files in your named file set.
Premium Feature — This feature is available in Premium Editions of LabKey Server. Learn more or contact LabKey.
Site Administrators of Premium Editions can disable file upload across the entire site. When activated, file upload to both the file and pipeline roots will be disabled.When file upload is disabled, the Upload button in the File Repository is hidden. Also, dragging and dropping files does not trigger file upload. Users (with sufficient permissions) can still perform the following actions in the File Repository, including: download, rename, delete, move, edit properties, and create folder.Note that attaching files to issues, wikis, or messages is not disabled by this setting.Changes to this setting are logged under the Site Settings Events log.To disable file upload site-wide:
This topic includes some common troubleshooting steps to take when you encounter issues with uploading and importing data via the pipeline or file browser.
Review the audit log for actions taking place when you encountered the problem.
For troubleshooting issues with uploading files in particular, a good first step is to check the permissions of the target directory in the file system directory itself. Check both the files directory where your site-level file root points (<LABKEY_HOME>/files by default) and the location where logs will be written (<LABKEY_HOME>/logs). LabKey Server (i.e. the 'labkey' user running the service) must have the ability to write files to these locations.
Pipeline Job Log File
When you import files or archives using drag and drop or other pipeline import methods, you will see the Data Pipeline import page. Status messages will show progress. Upon completion you may either encounter an error or notice expected information is missing.If there is an Error during import, the Status column will generally read "ERROR", though even if it does not, you may want to check these logs. There may be helpful details in the Info column. Click the value in the Status column for full details about the pipeline job.The Folder import page provides a Job Status panel, followed by a log file panel giving a summary of the full log. Scan the panel for details about what went wrong, typically marked by the word Error or Warning and possibly highlighted in red. If this information does not help you identify the problem, click Show Full Log File for the full file. You can also click links to View or Download the log file in the Job Status web part.
Files Not Visible
If you do not see the expected set of files in a Files web part, check the following. Some of these options are inherited by subfolders, so it may not be immediately clear that they apply to your context.
Is your Files web part showing a different named file set.
In a Premium Edition, the project may also be configured with file upload disabled.
You can check for unexpected settings at each level as follows:
Files Web Part: Click the (triangle) menu in the web part corner and select Customize. See if the expected file root is being displayed.
Folder: > Folder > Management > Files tab
Project: > Folder > Project Settings > Files
Site: > Site > Admin Console > Configuration > Files.
Import Data Button Not Available
If you are using a pipeline override for a folder that differs from the file directory, you will not see an Import Data button in the Files web part. You may either to change project settings to use the default site-level file root or you can import files via the Data Pipeline instead of the Files web part. To access the pipeline UI, go to: > Go to Module > Pipeline.
Project-level Files Not Moved When Site-wide Root is Changed
When the site-wide root changes for an existing installation, files in projects that use fileroots based on that site-level file root will be automatically moved to the new location. The server will also update paths in the database for all of the core tables. If you are storing file paths in tables managed by custom modules, the custom module will need register an instance of org.labkey.api.files.FileListener with org.labkey.api.files.FileContenService.addFileListener(), and fix up the paths stored in the database within its fileMoved() method.Files located in projects that use pipeline overrides or in folders with their own project- or folder-level file roots will not be moved by changing the site-level file root. If you have set project-level roots or pipeline overrides, files in these projects and their subfolders must be moved separately.
User Cannot Upload Files When Pipeline Override Set
In general, a user who has the Editor or Author role for a folder should be able to upload files. This is true for the default file management tool location, or file attachments for issues, wikis, messages, etc.The exception is when you have configured a folder (or its parent folders) to use a pipeline override. In that case, you will need to explicitly assign permissions for the pipeline override directory.To determine whether a pipeline override is set up, and to configure permissions if so, follow these steps:
Navigate to the folder in question.
Select > Go to Module > Pipeline.
Click Setup.
If the "Set a pipeline override" option is selected, you have two choices:
Keep the override and use the choices under the Pipeline Files Permissions heading to set permissions for the appropriate users.
Remove the override and use normal permissions for the folder. Select "Use a default based on the site-level root" instead of a pipeline override.
Adjust folder permissions if needed using > Folder > Permissions.
When paths or file names include any special characters, including but not limited to "%" and "+", check error logs for unexpected encoding of the path. Consistency changes have made pathnames more consistent, but if you have utilities or outside links depending on the previous special cased behavior, you may need to update them.
This topic defines commonly used terminology for files in LabKey Server.Root vs. directory. A root generally implies is the top level of inheritance throughout a tree of directories. A directory identifies just one spot in the file system.LabKey installation directory. The default directory in the file system for LabKey Server that contains folders for files, modules, the webapp, etc. Sometimes referred to as [LABKEY_HOME]. (Example: /labkey/labkey/ or C:\labkey\labkey\)Site-level file root. The directory in the LabKey Server's file system that contains your server's file directory structure (Example: /labkey/labkey/files). It can be set, typically at install time. This location is called a root, not just a directory, because it determines where the tree of file directories lives, not just the location of a single directory. The structure reflects your server's tree of projects and folders. Learn more here:
File directory. The specific location on the file system where files associated with a particular project or folder are placed. (Example: /labkey/labkey/files/project1/folder1/@files, where the folder of interest is folder1, a child of project1.) Learn more here:
File root. The directory in LabKey Server's file system that contains your project's file directory structure. (Example: /labkey/labkey/otherdata/project1/) This structure contains file directories in subfolders that match the structure of your project. Learn more here:
File root override. A custom destination for files for a given project. Can be set at the project level only. (Example: /labkey/labkey/otherdata/project1/myfiles). If a file root override is set, this root determines the the location of the tree of file directories for that project's subfolders. Learn more here:
Pipeline override. An explicitly set location for files that are subject to actions. Also determines the location where files are uploaded when uploaded via the pipeline UI. Allows security to be set separately vs. the default, folder-specific permissions. See Set a Pipeline Override.
Transfer Files with WebDAV
Use a WebDAV client as an alternative to the native LabKey Server interfaces for accessing files on LabKey Server. You can use a variety of options, including 3rd party clients like Cyberduck, to read, modify and delete files on the server.
When using OSX, you do not need to install a 3rd party WebDAV client. You can mount a WebDAV share via the dialog at Go > Connect to Server. Enter a URL of the form:
To have the URL generated for you, see the instructions above for Cyberduck.
<username%40domain.com> - The email address you use to log in to your LabKey Server, with the @ symbol replaced with %40. Example: Use username%40labkey.com for username@labkey.com
<www.sitename.org> - The URL of your LabKey Server. Example: www.mysite.org
<projectname> - The name of the project you would like to access. If you need to force a login, this project should provide no guest access. Example: _secret
Linux WebDAV Clients
Available 3rd party clients:
Gnome Desktop: Nautilus file browser can mount a WebDAV share like an NFS share.
KDE Desktop: Has drop down for mounting a WebDAV share like an NFS share.
Premium Feature — Available in the Professional and Enterprise Editions of LabKey Server. Also available as an Add-on to the Starter Edition. Learn more or contact LabKey.
LabKey Server can integrate cloud storage for management of large data files using Amazon S3 (Simple Storage Service). Support for other storage providers will be considered in the future. For more information about this feature and possible future directions, please contact LabKey.Cloud storage services are best suited to providing an archive for large files. Files managed by cloud storage can be exposed within LabKey as if they were in the local filesystem. You can add and remove files from the cloud via the LabKey interface and many pipeline jobs can be run directly on them. This section outlines the steps necessary to access data in an S3 bucket from the Files web part on your server.
Cloud Services offer the ability to upload and post large data files in the cloud, and LabKey Server can interface with this data allowing users to integrate it smoothly with other data for seamless use by LabKey analysis tools. In order to use these features, you must have installed the cloud module in your LabKey Server.Contact your Account Manager for assistance if needed.
Buckets
Cloud Storage services store data in buckets which are typically limited to a certain number by user account, but can contain unlimited files. LabKey Server Cloud Storage uses a single bucket with a directory providing a pseudo-hierarchy so that multiple structured folders can appear as a multi-bucket storage system.Learn more about Amazon S3 Buckets here: Working with Amazon S3 Buckets
Encryption and Security
The Cloud module supports AWS S3 "default" AES encryption. This can be configured when the bucket is provisioned. With "default" encryption S3 transparently encrypts/decrypts the files/objects when they are written to or read from the S3 bucket.AWS S3 also supports unique KMS (Key Management System) encryption keys that are managed by the customer within AWS. Other security controls on the S3 bucket such as who can access those files via other AWS methods is in full control of the customer by using AWS IAM Policies.Regardless of how the S3 buckets are configured, LabKey has a full authentication and authorization system built in to manage who can access those files within LabKey.
Set Up Cloud Storage
To set up to use Cloud Storage with LabKey Server, you need the targets on AWS/S3, and configuration at the site-level before you'll be able to use the storage.
These Storage Configs will then be available in your projects and folders as Cloud Stores.
Use Files from Cloud Storage
Once you have completed the setup steps above, you will be able to select which Cloud Stores to use on a per-folder basis as required. The enabling and usage are covered in this topic:
If you are interested in learning more about, or contributing to, the future directions for this functionality, please contact LabKey.
AWS Identity Credentials
Premium Feature — Available in the Professional and Enterprise Editions of LabKey Server. Also available as an Add-on to the Starter Edition. Learn more or contact LabKey.
The identity and credential LabKey will use to access your S3 bucket are generated by creating an AWS Identity .
On the AWS console click "Add User", provide a user name, select Programmatic Access, create a new group and give it AdministratorAccess. If AdministratorAccess is not possible, the detailed permissions required are listed below.At the end of the AWS setup wizard, you will be given an "Access key id" and a "Secret access key". Enter these in the Identity and Credentials fields when you create a Cloud Account on LabKey Server.
S3 Permissions Required
The detailed permissions required for S3 access are listed below. Substitute your bucket name where you see BUCKET_NAME.
Additionally, if ACLs are defined on individual objects within a bucket, the user will need READ and READ_ACP permission to each object for read-only usage, and WRITE and WRITE_ACP for write usage.See more information about S3 permissions in the AWS documentation.
Premium Feature — Available in the Professional and Enterprise Editions of LabKey Server. Also available as an Add-on to the Starter Edition. Learn more or contact LabKey.
This topic outlines how to create and configure S3 cloud storage on LabKey Server. Each bucket on S3 that you plan to access will be defined as a Storage Config on LabKey Server, accessed through a Cloud Account. You will then be able to select which storage config to use on a per-folder basis.
Before you can use your Cloud Storage account from within LabKey Server, you must first create the bucket you intend to use and the user account must have "list" as well as "upload/delete" permissions on the bucket.
It is possible to have multiple cloud store services per account.
AWS S3 "default" AES encryption is supported and can be configured on the S3 bucket when the bucket is provisioned.
With "default" encryption S3 transparently encrypts/decrypts the files/objects when they are written to or read from the S3 bucket.
AWS S3 also supports unique KMS (Key Management System) encryption keys that are managed by the customer within AWS.
Create Cloud Account On LabKey Server
To access the bucket, you create a cloud account on your server, providing a named way to indicate the cloud credentials to use.
Select > Site > Admin Console.
Under Premium Features, click Cloud Settings.
If you do not see this option, you do not have the cloud module installed.
Under Cloud Accounts, click Create Account.
Enter an Account Name. It must be unique and will represent the login information entered here.
Select a Provider.
Enter your Identity and Credential. See AWS Identity for details.
Click Create.
This feature uses the encrypted property store for credentials and requires an administrator to provide an encryption key in the application.properties file. LabKey will refuse to store credentials if a key is not provided.
Create Storage Config (on LabKey Server)
Next define a Storage Config, effectively a file alias pointing to a bucket available to your account. LabKey can create new subfolders in that location, or if you want to use a pre-existing S3 subdirectory within your bucket, you can specify it using the S3 Path option.
Click Create Storage Config on the cloud account settings page under Cloud Store Service.
If you navigated away, select > Site > Admin Console. Under Premium Features, click Cloud Settings.
Provide a Config Name. This name must be unique and it is good practice to base it on the S3 bucket that it will access.
Select the Account you just created from the pulldown.
Provide the S3 Bucket name itself. Do not include "S3://" or other elements of the full URL with the bucket name in this field. Learn more about bucket naming rules here
Select Enabled.
If you disable a storage config by unchecking this box, it will not be deleted, but you will be unable to use it from any container until enabling it again.
S3 Path: (Optional) You can specify a path within the S3 bucket that will be the configuration root of any LabKey folder using this configuration. This enables use of an existing folder within the S3 bucket. If no path is specified, the root is the bucket itself.
Directory Prefix: (Optional) Select whether to create a directory named <prefix><id> in the bucket or S3 path provided for this folder. The default prefix is "container".
If you check the Directory Prefix box (default), LabKey will automatically create a subdirectory in the configuration root (the bucket itself or the S3 path provided above) for each LabKey folder using this configuration. For example, a generated directory name would be "container16", where 16 is the id number of the LabKey folder. You can see the id number for a given folder/container by going to Folder > Management > Information, or by querying the core.Containers table through the UI or an API. You may also find the reporting in Admin Console > Files helpful, as it will let you navigate the container tree and see the S3 URLs including the containerX values. Note that using this option means that the subdirectory and its contents will be deleted if the LabKey folder is deleted.
If you do not check the box, all LabKey folders using this configuration will share the root location and LabKey will not delete the root contents when any folder is deleted.
SQS Queue URL: If your bucket is configured to queue notifications, provide the URL here. Note that the region (like "us-west-1" in this URL) must match the region for the S3 Bucket specified for this storage config.
Click Create.
Authorized administrators will be able to use the Edit link for defined storage configs for updating them.
Configure Queue Notifications for File Watchers
If you plan to use file watchers for files uploaded to your bucket, you must first configure the Simple Queue Service within AWS. Then supply the SQS Queue URL in your Storage Config on LabKey Server.Learn more in this topic:
Premium Feature — Available in the Professional and Enterprise Editions of LabKey Server. Also available as an Add-on to the Starter Edition. Learn more or contact LabKey.
Once you have configured S3 cloud storage on your LabKey Server, you can enable and use it as described in this topic.
In each folder or project where you want to access cloud data, configure the filesystem to use the appropriate cloud storage config(s) you defined. Cloud storage at the project level can be inherited by folders within that project, or folders can override a project setting as needed.
Note that if a cloud storage config is disabled at the site-level it will not be possible to enable it within a folder or project.
Navigate to the folder where you want to enable cloud storage and open > Folder > Management.
If you want to enable cloud storage at the project level, open > Folder > Project Settings instead.
Select the Files tab.
Under Cloud Stores, the lower section of the page, first enable the desired cloud stores using the checkboxes.
Note that it's possible to disable a cloud storage config at the site level. If a config is not enabled at the site level, enabling it in a folder or project will have no effect.
Click Save.
After saving the selection of cloud stores for this container, you will be able to select one in the file root section higher on this page.
Under File Root select Use cloud-based file storage and use the dropdown to select the desired cloud store.
If you select this option before enabling the cloud store, you will see an empty dropdown.
Existing Files: When you select a new file root for a folder, you will see the option Proposed File Root change from '<prior option>'. Select what you want to happen to any existing files in the root. Note that if you are not using directory containers, you will not be able to move files as they will not be deleted from the shared root. See Migrate Existing Files for details about file migration options.
Click the Save button a second time.
Configure a Files Web Part
Once you have configured the file root for your folder to point to the cloud storage, you will access it using the usual Files web part.
Go to the Files web part in your folder.
If you don't have one, select > Page Admin Mode.
From the dropdown in the lower left, click <Select Web Part> and choose Files.
Click Add.
If desired, you can give the web part a name that signals it will access cloud storage:
To do so, choose Customize from the (triangle) menu in the Files web part header.
Enter the Title you would like, such as "Cloud Files".
Checking the box to make the Folder Tree visible may also be helpful.
You can also give the webpart a descriptive title while you are in the customize interface.
Click Submit.
The Files web part will now display the cloud storage files as if they are part of your local filesystem, as in the case of the .fcs file shown here:The file is actually located in cloud storage as shown here:
When a download request for a cloud storage file comes through LabKey Server, the handle is passed to the client so the client can download the file directly.
When a file is dropped into the Files web part, it will be uploaded to the cloud bucket.
Files uploaded to the S3 bucket independently of LabKey will also appear in the LabKey Files web part.
Run Pipeline Tasks on S3 Files
Once you have configured an S3 pipeline root, folders can be imported or reloaded from folder archives that reside in cloud storage.
Import folder (from either .folder.zip archive, or folder.xml file)
Use File Watchers with Cloud Files
To use File Watchers, you will need to enable SQS Queue Notifications on your AWS bucket, then include that queue in the storage config. Learn more in this topic:
Once configured, you can Reload Lists Using Data File from the cloud. Other file watcher types will be added in future releases.
Delete Files from Cloud Storage
If you have configured cloud storage in LabKey to create a new subdirectory (using Directory Prefix) for each new folder created on LabKey, the files placed within it will be associated with the LabKey folder. If you delete the LabKey folder, the associated subfolder of your S3 bucket (and all files within it) will also be deleted.If you instead configured cloud storage to use an existing S3 folder, any files placed there will be visible from within the LabKey folder, but will NOT be deleted if the LabKey folder is deleted.
Premium Feature — Available in the Professional and Enterprise Editions of LabKey Server. Also available as an Add-on to the Starter Edition. Learn more or contact LabKey.
If you plan to use file watchers for files uploaded to your bucket, you need to supply an SQS Queue URL in your Storage Config on LabKey Server. This topic describes how to configure the bucket and queue on AWS to support this.
You will be configuring an SQS Queue that LabKey Server can 'watch' for notifications, then setting up event notifications on the bucket that will add to that queue when certain files are added to certain bucket locations.
Create SQS Queue on AWS
First, you must configure a Simple Queue Service within AWS to which your bucket will be able to post notifications.If you have not already set up your bucket, follow the instructions in this topic: Configure Cloud StorageTake note of the region (like "us-west-2") for your bucket.
In your AWS account, click Create Queue.
Select Standard queue. (FIFO queues are not supported).
Accept the default configuration options.
Disable Server-side encryption. The file watcher implementation is not compatible with signature version 4 for the SQS API that encryption requires.
Set the Access Policy for the queue to Advanced and provide for both of the following:
Allow users and IAM roles to manipulate the queue/messages (create, read, update, delete). The example below allows full control. This is what will allow the credentials supplied to LabKey Server to read/delete the messages from the Queue.
Once you've enabled the queue on AWS, create or edit the LabKey Storage Config for this bucket, providing the SQS Queue URL.Note that the region (like "us-west-2" in this URL) must match the region for the S3 Bucket specified in this storage config.
Configure S3 Event Notifications
Next, configure your bucket to send event notifications to your SQS Queue.
Log in to AWS and access the bucket you intend to monitor.
On the bucket's Properties tab, click Create Event Notification.
Under General configuration, name the event (i.e. MyFileDrop) and provide optional prefixes and suffixes to define which bucket contents should trigger notifications. Only objects with both the specified prefix and suffix will trigger notifications.
Prefix: This is typically the directory path within the bucket, and will depend on how you organize your buckets. For example, you might be using a "cloud_storage" main directory and then employ a LabKey directory prefix like "lk_" for each LabKey folder, then a "watched" subdirectory there. In this case the prefix would be something like "/cloud_storage/lk_2483/watched/". This prefix should either match, or be a subdirectory of, the Location to Watch you will set in your file watcher.
Suffix: A suffix for the files that should trigger notifications, such as ".list.tsv" if you want to look for files like "FirstList.list.tsv" and "SecondList.list.tsv" in the container specified in the prefix.
Event Types:
Select All Object Create Events.
Destination:
Select SQS Queue, then either:
Choose from your SQS Queues to use a dropdown selector or
Enter SQS queue ARN to enter it directly.
Click Save when finished.
Now when files in your bucket's Prefix location have the Suffix you defined, an event notification will be triggered on the queue specified. LabKey is "monitoring" this queue via the Storage Config, so now you will be able to enable file watchers on those locations.
Note that the files in your bucket that trigger notifications do not have to be the same as the files your file watcher will act upon. For example, you might want to reload a set of list files, but only trigger that file watcher when a master manifest was added. In such a scenario, your event notifications would be triggered by the manifest, and the filewatcher would then act on the files matching it's definition, i.e. the lists.
Troubleshooting Event Notifications
If you have multiple event notifications defined on your queue, note that they will be validated as a group. If you have existing notifications configured when you add a new one for your LabKey file watcher, you will not be able to save or edit new ones until you resolve the configuration of the prior ones. When in a state that cannot be validated, notifications will still be sent, but this is an interaction to keep in mind if you encounter problems defining a new one.
Create File Watcher for Cloud Files
Once cloud storage with queue notifications is configured, and enabled in your folder, you will be able to create a Files web part in the folder that "surfaces" the S3 location. For example, you can drop files into the bucket from the LabKey interface, and have them appear in the S3 bucket, or vice versa.You will also be able to configure file watchers in that folder triggered by the event notifications on the bucket.The Reload Lists Using Data File task is currently supported for this feature. Other file watcher types will support S3 cloud loading in future releases.
A workgroup or user community can use LabKey message boards to post announcements and files and to carry on threaded discussions. A message board is useful for discussing ongoing work, answering questions and providing support, and posting documents for project users. Message boards can also be used to track and document workflow task completion and coordinate handoff of responsibility for different steps when a more involved issue tracking system is not warranted.
Message boards let you post team announcements and set up threaded conversations among users. They can be made available to the general public or restricted to a select set of users. Announcements might include planned site outage announcements, new results in a study, organizational and personnel changes, introducing new features for users, etc.Individual announcements in the message board can also include threaded discussions contributed by users for asking questions and receiving public (or private) answers. This topic covers creating and using a basic message board.
To use this topic as a tutorial, it is good practice to use a new folder where you have administrator access and it is suitable to do testing. For example:
Log in to your server and navigate to your "Tutorials" project, "Collaboration Tutorial" subfolder. Create them if necessary, accepting default types and settings.
If you don't already have a server to work on where you can create projects and folders, start here.
If you don't know how to create projects and folders, review this topic.
Create Message Board
Begin in the folder where have administrator permissions and you want to create the message board. Each container can only contain a single board for messages. If you need separate sets of conversations, place them in different folders.
If your page does not already have a Messages web part:
You can use the (triangle) web part menu to move it up the page if you like.
Click Exit Admin Mode.
Learn more about the options available on the (triangle) menu in this topic.
Post New Messages
To post a new message, the user must have the "Author" role or higher. Users without the necessary permission will not see the New button.
In the Messages web part, click New.
Enter the Title(Required).
Enter the Body of the message. The default format is Markdown. You can simply type into the "Source" tab, or use Markdown formatting syntax. A brief syntax guide is included at the bottom of the screen. Use the "Preview" tab to see what your message will look like.
Use the Render As drop-down menu to change how the body text can be entered (if you don't see this option, an administrator has disabled the format picker). Options are:
To add an attachment, click Attach a file. Attachments should not exceed 250MB per message.
Click Submit to post your message.
Additional fields may be available if your administrator chose to include them.If you are using this topic as a tutorial for learning to use a new message board, create a few messages and reply to a few of them so that you have some content to work with in the next sections.
View, Respond-to, Edit, and Delete Messages
Message board users with authorization can post new messages, respond to continue conversations, subscribe to conversations, and configure their own notification preferences. Users with Editor or higher permissions can also edit existing messages.The beginning of the message text is displayed in the Messages web part - you can expand the text within the web part by clicking More. To open the message for other features, click View Message or Respond. You can also open the message by clicking the title in a Message List web part.The original message is followed by any responses; all are marked with the username and date of the post. Click Respond to add a response message of your own. If you have sufficient permissions, you will also have one or more of these additional options:
Edit: each message you have written shows an edit link if you need to make corrections. If you are an administrator, you can edit messages written by others.
View list: view the messages on this board in list format
Delete Message: Delete the original message and all replies.
Subscribe and Unsubscribe to Messages
When you add a message, you are automatically subscribed to receive responses to a message when you post it. In the message view shown above, you can click Unsubscribe if you do not want to be notified of responses. When viewing another user's posted message, click Subscribe in the upper right to enable notifications for the specific thread, or the entire forum. Choosing "forum" will take you to the page for configuring email preferences.
Configure Email Preferences
The message board administrator can specify default email notification policy for the project or folder. Each user can choose their own notifications, potentially overriding the default settings, as follows:To set your email preferences:
Return to the Messages web part by clicking the [Folder Name] link near the top of the page.
Click the (triangle) at the top right of the Messages web part.
Select Email Preferences.
Choose which messages should trigger emails (none, mine, or all) and whether you prefer each individual message or a daily digest.
Check the box if you want to Reset to the folder default setting.
You can also enable notifications for specific conversations or for the message board as a whole using the Subscribe option while viewing a specific message.
Additional Message Fields (Optional)
The following message fields are turned off by default but you will see them if they have been activated by an administrator: Status, Assigned To, Notify, and Expires.Status: Track whether messages are "Active", i.e. requiring further work or attention, or "Closed". Once a message is closed, it is no longer displayed on the Messages or Messages List web parts.Assigned To: Assign an issue to a particular team member for further work, useful for sharing multi-part tasks or managing workflow procedures.Notify: List of users to receive email notification whenever there are new postings to the message. The Notify field is especially useful when someone needs to keep track of an evolving discussion.Expires: Set the date on which the message will disappear, by default one month from the day of posting. Once the expiration date has passed, the message will no longer appear in either the Messages or Messages List web parts, but it will still appear in the unfiltered message list. You can use this feature to display only the most relevant or urgent messages, while still preserving all messages. If you leave the Expires field blank, the message will never expire.
A message board holds the set of messages created in a given folder container. Creating and using message boards themselves is covered in the topic Use Message Boards.This topic describes how administrators can configure message boards to best serve the needs of the working group using them. Match the terminology, notifications, and displays to what your group expects.
Message Board vs. Message Web Parts: The message board itself is an underlying set of messages created in a folder; there can only be one per folder. However, there are two web parts (user-interface panels) available for displaying a given message board in different ways. You could include multiple web parts in a folder to display messages, but they would all show the same underlying set of messages. Learn more about message web parts below.Message Board Admin: The Admin option, available on the (triangle) menu of either web part when in in page admin mode, gives you access to administration features such as changing the name of the board, the fields available for users creating messages, and other options.Messages Web Part Customization: The Customize option, available on the (triangle) menu of the Messages web part lets you customize the display in the web part. It does not change the underlying message board.
Message Board Admin
To configure settings, or "administer" a message board, use page admin mode to enable the Admin option:
Enter > Page Admin Mode.
Click the (triangle) menu and choose Admin.
"Admin" takes you to the "Customize [Name of Message Board]" page.
The options are described below.
Board name: The name used to refer to the entire message board. Examples: "Team Discussions," "Building Announcements," etc.Conversation name: The term used by your team to refer to a conversation. Example: "thread" or "discussion."Conversation sorting: Whether to sort by initial post on any given conversation (appropriate for announcements or blogs) or most recent update or response (suitable for discussion boards).Security:
Off (default): Conversations are visible to anyone with read permissions, content can be modified after posting, and content will be sent via email (if enabled by users).
On with email: Only editors and those on the notify list can view conversations, content can't be modified after posting, content is sent via email (if enabled by users).
On without email: Only editors and those on the notify list can view conversations, content can't be modified after posting, content is never sent via email.
Moderator review: Adding moderator review can help reduce "noise" messages, particularly with message boards that are generally open to the public (e.g., when self-sign-on is enabled and all site users are assigned the "Author" role). Messages from users with the "Author" or "Message Board Contributor" role are subject to review when enabled. (Messages from Editors and Admins are never subject to moderator review.) Options:
None: (Default) New messages never require moderator review.
Initial Post: Authors and Message Board Contributors must have their initial message(s) approved by a moderator. Subsequent messages by approved users are approved automatically.
New Thread: All new message threads started by an Author (or Message Board Contributor) must be approved by a moderator.
All: Every new message from an Author (or Message Board Contributor) must be approved by a moderator.
Allow Editing Title: Check to enable editing the title of a message.Include Notify List: Check to enable the Notify field, allowing you to list users to notify about the conversation. On a secure message board, you can use this feature to notify a user who does not have editor permissions on the message board itself.Include Status: Enables a drop-down for the status of a message, "active" or "closed". When enabled, any user who can edit a message (admins, editors, and the message's author) can also change it's status. Note that a closed message is still available for new responses.Include Expires: Allows the option to indicate when a message expires and is no longer displayed in the messages web part. The default is one month after posting. Expired messages are still accessible in the Message List.Include Assigned To: Displays a drop-down list of project users to whom the message can be assigned as a task or workflow item. You can specify a default assignee for all new messages.Include Format Picker: Displays a drop-down list of options for message format: Wiki Page, HTML, or Plain Text. If the format picker is not displayed, new messages are posted as plain text.Show poster's groups, link user name to full details (admins only): Check this box to provide admins with details about the group memberships of whoever posted each message and a link directly to their profile. Other users will see only the user name.Email templates: Click links to:
Customize site-wide template. You can also select Email > Site-Wide Email Template directly from the messages web part menu.
Customize template for this project (or folder). You can also select Email > Project (or Folder) Email Template directly from the messages web part menu.
Example: Configure Message Board
As an example, you could make the following changes to a basic message board, as you created in the topic Use Message Boards.
Enter > Page Admin Mode.
Click the (triangle) menu and choose Admin.
Board Name: Replace "Messages" with "Our Team Message Board"
Conversation name: Replace "Message" with "Announcement".
Click Save.
The message board will now look like this:
Message Board Security Options
Consider security and notification settings for your message boards when defining them. There are two basic types of message board, depending on the Security setting.
For a basic message board with Security=OFF, the following policies apply:
Author/Message Board Contributor: Add new messages and edit or delete their own messages.
Editor/Admin: Can also edit any message posted to the message board, even if it was posted by someone else.
Email will be sent when users are on the "Notify List" for a message, or subscribe to that individual message, or subscribe to the board as a whole.
A Secure Message Board, where Security is set to "On with email" or "On without email", applies these policies of limited access:
No message content may be edited after posting.
As with an ordinary message board, users can control whether they receive email for the "On with email" option. They cannot elect to receive email if "On without email" is set.
Users must be logged in to view messages and have sufficient permissions as follows:
Author/Message Board Contributor: Can still add new messages, but cannot read messages unless they are on the "Notify List" for that message.
Editor/Admin: Can read messages, but cannot edit messages. To add a user to the message list, they can reply to a message and include the user on the message list in the reply.
The announcements schema will be unavailable in the schema browser, to prevent queries from exposing secure messages.
Learn more about roles in this topic: Security Roles Reference
Learn more about email notifications below.If a board is configured to email all users, you may want to restrict posting access or implement a moderator review process.You may also choose to restrict the visibility of the web part based on permissions, but note that doing so does not prevent a user with access to the folder from finding and using it, such as by using the URL.
Moderator Review
When moderator review is enabled, an administrator must approve messages written by any users with the "Author" or "Message Board Contributor" role. There are three levels of moderation available:
Initial Post: Only the first post by a given user must be approved by a moderator before that user is approved to reply and post new messages.
New Thread: Replies to existing threads will not require approval, but every new thread will.
All: All messages, both new and reply, must be approved by a moderator to be posted.
Only after a message is approved by a moderator will it be made visible, indexed for full-text search, sent in individual email notifications, or included in daily digest emails. Messages from Editors and Administrators are always posted without moderator review.Once enabled, the Messages > Admin menu will include a link to the Moderator Review Page. Note that you have to save the enabled setting for this link to be shown.All messages posted to the board requiring review are initially saved, but with the "Approved" column set to false, so they are not posted or included in daily digests.Administrators that are subscribed to get individual emails for the board will be sent an email notification that moderator review is required, including a link to the Moderator Review Page and details about the new message. These notifications are not sent to administrators subscribed to receive daily digests for the board. When you enable moderator review, check to make sure that the intended moderator(s) will receive notifications.On the Moderator Review Page, any moderator can review the messages and select rows to either:
Approve: The message is posted and notifications sent.
Mark As Spam: The message is marked as rejected and the user is blocked from posting.
Note: If you edit an existing message board to enable moderator review of the initial post, all users who have posted messages in the past are already approved.
Email Notifications for Messages
Email notification is always disabled when using a secure message board with Security="On without email". This section applies only to message boards where email can be sent.
Project administrators can set the defaults for the message board's email notification behavior for individual users or for all users at the folder level. Any user can override the preferences set for them if they choose to do so.
In > Page Admin Mode.
Click the (triangle) in the Messages web part.
Select Email > Administration to open the folder notifications tab.
In some cases you may want to set up a mechanism for emailing periodic broadcast messages to all users, such as notification of a server upgrade affecting all projects. You would probably want to limit the permission to send such wide broadcasts to only users who would need to do so. Individual users can still opt-out of these email notifications.
Create a folder for your broadcast announcements and grant Read permissions to All Site Users.
Grant the Author role in this folder to individuals or groups who should be able to send broadcast messages.
Set Default Setting for Messages to "All conversations."
Create a new message. Notice that there is an alert above the Submit button indicating the number of people that will be emailed.
If you only need to send a one time email to all users, another approach is to open Admin > Site > Site Users, export the list to Excel, copy the contents of the Email column, and paste into a new email message.
Message Board Web Parts
(1) The Messages web part displays the first few lines of text of the most recent messages. Each message is labeled with its author and the date it was posted, and includes a link to view or respond to the message. Messages longer than the lines displayed will have a More button to expand them. This is the default and primary view of a message board available in many new folders by default.(2) The Messages List web part displays a sortable and filterable grid view of all messages, as shown below.The Messages web part is the primary way to view a message board. When you are not in page admin mode, a limited set of user options are available on the menu. Note that the "Customize" option on the web part menu lets you customize the web part itself as described below, and is not used to change message board configuration.As with all web parts, when you are in Page Admin Mode, you gain additional options on the web part (triangle) menu, such as moving web parts up or down the page. You can also control which users can see web parts in this mode. See Web Parts: Permissions Required to View for more information.When not in page admin mode:
New: Create a new message
View List: Switch to the Messages List view of the message board.
In page admin mode, the following additional options are added:
Email: With these submenu options:
Preferences: Same as the Email Preferences menu item when not in page admin mode.
Administration: Configure notification settings for the folder and individual users.
Site-Wide Email Template: Edit the template for messages that is applied by default anywhere on the site that doesn't define a project or folder email template.
Project (or Folder) Email Template: Edit the template used for messages in this folder or project. This will override any site-wide template.
Admin: Edit the message board settings.
Permissions: Control permissions needed to see the web part.
Move up/down: Adjust the position on the page.
Remove from page: Note that this removes the user interface, not the underlying message board.
Hide frame: Hide both the box outline and web part menu from users not in page admin mode.
Customize the Messages Web Part
You can switch the display in the web part between a simple message presentation and full message display by selecting Customize from the Messages menu.
Customize the Messages List Web Part
The Messages List is an alternative user interface option, providing a grid view of all the messages in the current board. It can be filtered and sorted, new messages can be added by clicking New and the contents of any existing message can be accessed by clicking the conversation title.Like the Messages web part, the options available on the web part menu vary depending on whether you are in page admin mode.When not in page admin mode:
A wiki is a hierarchical collection of documents that multiple users can edit (with adequate permissions). Individual wiki pages can be written in HTML, Markdown, Wiki-syntax, or plain text. On LabKey Server, you can use a wiki to include formatted content in a project or folder. You can embed other wikis, webparts, and even live data in this content.
Wiki Admin Guide: Learn how to set up and manage a wiki. (Admin Topics)
Wiki User Guide: Learn how to write, edit and manage individual pages.
Wikis provide an easy way to create web pages and arrange content for users. Pages rendered with different syntax, such as Markdown, HTML, Wiki syntax and even plain text can be used in combination.
To use this topic as a tutorial, you can use any folder where you have administrator access and it is suitable to do testing. For example:
Log in to your server and navigate to your "Tutorials" project, "Collaboration Tutorial" subfolder. Create them if necessary, accepting default types and settings.
If you don't already have a server to work on where you can create projects and folders, start here.
If you don't know how to create projects and folders, review this topic.
Example Page
The following example wiki page is shown as an administrator would see it.
The page includes a title in large font.
Links are shown for editing the page, creating a new one, viewing the history of changes, etc. These buttons are shown only to users with appropriate permissions.
Links and images can be included in the body of the page.
The following steps will take you through the mechanics of creating and updating wiki pages. As a resource to use, first download this image file and keep the name demoImage.png to use our examples as written:
The default wiki page format is HTML, which is powerful, but not needed for most wiki pages. Wiki syntax is a much simpler format to read and write than HTML, and supports many useful layout and formatting options.
Navigate to the folder where you want to create the page.
In the Wiki web part, click Create a new wiki page.
If this link is not present, i.e. another wiki already exists in the folder, use the menu for the wiki web part and select New.
In the wiki editor, if the Body section indicates a format other than Wiki Page, click Convert To....
In the Change Format pop up dialog, select Wiki Page and click Convert.
In the New Page form, make the following changes:
In the Name field, leave "default" if this is the first page, or enter a new unique page name like "example".
In the Title field, paste "Correlation Between CD4 and Lymphs" to match our example.
Copy the following and paste into the Body field:
1 Results
See the section [detailed results|#more] for a full explanation of the graph below.
1.1 Figure 1
[demoImage.png]
{anchor:name=more} **Figure 1** shows the mean count values for both cohorts at initial subject evaluation. The correlation coefficient between Lymphocyte and CD4 cell counts was significant (0.7). There were 33 subjects in one cohort and 78 subjects in the other cohort.
Uncheck the box for "Show Attached Files". When this box is checked, you will see a listing of all attached file names at the bottom of your page, which can be useful for attachments a user should download, but is unnecessary and potentially confusing when the attached files are images displayed within the text.
Scroll down and click Attach a file. Click Choose File (or Browse) and select the demoImage.png file you downloaded.
You may notice that you see a Pages panel (the wiki table of contents) appears both on the folder home page and on the display page for an individual wiki page.
Next we'll add with a short text-only wiki using the Markdown rendering option.
In the upper corner of any wiki web part, select New from the (triangle) menu.
In the New Page form, make the following changes:
In the Name field, enter "projectx".
In the Title field, enter "Project X".
Click the Convert To... button on the right, select Markdown, then click Convert.
In the Body panel, enter the following text: "Project X data collection proceeds apace, we expect to have finished the first round of collection and quality control by...".
Notice a brief guide to additional Markdown formatting options appears at the bottom of the editing pane. For more information, see Markdown Syntax.
Click Save & Close.
After saving this second wiki, you will be viewing it on a page of it's own. Unlike the first page you created, this one was not automatically added to a web part on the home page of the folder.
Notice that you now have a link near the top of the page Collaboration Tutorial. You can click it to return to the main folder page. The Pages web part (the table of contents) in the right hand column shows a link to your new page.
Create an HTML Page
HTML formatting can provide additional control over the layout and rendering of any wiki page. You can use either the visual tab or source tab where you can directly edit the HTML source. As a way of comparing to the wiki syntax above, we'll show how the same page would look using HTML syntax, but with another page as the 'detailed results' link target.
In the Pages panel, either on the main page or while viewing an individual page, click the , and select New.
In the New Page form, make the following changes:
In the Name field, enter "projectz".
In the Title field, enter "Project Z".
Click Convert To....
In the Change Format pop up dialog, select HTML (if it is not selected by default), and click Convert.
Notice the Body section now has two tabs: Visual and Source. Use the Source tab for this example.
Paste the following into the body panel:
<h3 class="heading-1">Results</h3> <p>See the section <a href="projectx">detailed results</a> for a full explanation of the graph below.</p> <h3 class="heading-1-1">Figure 1</h3> <p><img src="demoImage.png" alt="" /></p> <p><strong class="bold">Figure 1</strong> shows the mean count values for both cohorts at initial subject evaluation. The correlation coefficient between Lymphocyte and CD4 cell counts was significant (0.7). There were 33 subjects in one cohort and 78 subjects in the other cohort.</p>
Scroll down and click Attach a file. Click Choose File (or Browse) and select the demoImage.png file you downloaded.
Uncheck the box for "Show Attached Files". When this box is checked, you will see a listing of all attached file names at the bottom of your page, which can be useful for attachments a user should download, but is unnecessary and potentially confusing when the attached files are images displayed within the text.
Click Save & Close.
You have now recreated the example wiki shown at the top of this tutorial page. The link in this case only goes to the "Project X" wiki you created; in practice you could direct this link to any other location where your detailed results were located.
Organize Wiki Pages
Now that the menu on the far right (Pages) contains three links to the new pages, we can show how you might organize and manage multiple pages. To simplify things, first edit the "Correlation Between CD4 and Lymphs" page so that the title is to "Project Y".When you are viewing a page, the title is bolded on the Pages list. As you add pages, links are added to the menu creating a table of contents. You can rearrange their order, and can also arrange your table of contents hierarchically, making some pages "parents", with "children" pages underneath.
In the Pages web part, click Project X.
Click Manage in the wiki header.
All sibling pages can be reordered with the Move Up/Move Down buttons, not just the page you selected. Any changes will not be reflected in the Pages web part until you save.
Using the Parent dropdown, select "Project Z (projectz)". This makes Project Z the parent topic for Project X. Notice the "Project X" topic no longer shows any siblings (as "Project Z" has no other child pages).
Click Save.
Notice the Pages web part: any parent page now has a or for expanding and collapsing the table of contents.
For a more elaborate example, browse the LabKey documentation table of contents on the right of the page you are reading now.Learn more about managing wiki pages in this topic: Manage Wiki Pages
Include Links in Wikis
Wiki pages can incorporate links to other pages in the same wiki, wiki pages in other containers on the site, outside websites, and for some formats, on page anchors can provide inter-page links.
Inter-page Links
To link to content on the same page, such as to create a table of contents at the top of a topic, you can use either Wiki syntax or HTML. In both cases you need to place a uniquely-named anchor somewhere, then use # syntax to link to it.
In Wiki syntax:
{anchor:name=target} This is the target text I want to jump to.
And elsewhere, I can [link to that text|#target].
In HTML:
<p id="target">This is the target text I want to jump to.</p>
<p> And elsewhere, I can <a href="#target">link to that target</a>.</p>
The implementation of Markdown LabKey uses does not support on-page anchors.
Link to Local Page
To link to a page in the local wiki, you need only name it:
In Wiki syntax, use this to use the page title as display text:
[myPageName]
In Wiki syntax, use this for the same link with non-title display text:
[Display text|myPageName]
In HTML, use something like:
<a href="myPageName">Display text</a></li>
Link to Anchor on Another Page
Combine the above two to link to an anchor on another page.
In wiki syntax:
[Display text|myPageName#target]
In HTML, you can also use controller-action syntax. This is required if linking to an on-page anchor on another page:
All changes to wiki pages are tracked and versioned, so you can see how a document has developed and have a record of who made changes. To see these features, first make an edit to one of your pages, in order to build an editing history for the page.
In the Pages menu, click Project X.
Click Edit. (If you do not see the edit link, you may need to log in again.)
Make some changes. For example, you might add some statement about "details" from Project Z being here (to support the link from the Project Z wiki we included).
Click Save & Close.
Click History to see the list of edits.
Click Version 1. You'll see the original page.
Click Compare With... and then select Latest Version (your_username) to see the changes made between the two revisions. Green highlighting shows added content; anything removed would be highlighted in pink.
This Wiki Admin Guide covers how administrators can set up and administer wikis and wiki web parts. To learn how to use a wiki once you have set one up, please read the Wiki User Guide.
Folders of any type can contain wikis, provided the module is enabled. The simplest folder type to use if you are simply setting up a set of documentation pages is "Collaboration".
In the folder where you want to create a wiki, enter > Page Admin Mode.
From the Select Web Part pulldown, select Wiki and click Add.
If there is already a page named "default" in the current container, it will be shown, otherwise you will see options to get started:
Click Create a new wiki page to create your first page, named "default" by default.
Enter any Name and Title you like. If you leave "default" as the name, it will display in your new wiki webpart by default.
Notice the wiki "Type" in the Body section. If it is "HTML", consider starting with one of the easier-to-use syntax options available, including Wiki syntax and Markdown.
Click Convert To... and choose Wiki Page in the modal. Click Convert.
Type your content in the Body section.
Click Save & Close.
If you left the page Name as "default" you will see the web part populated with your new wiki content.
If you changed the page name, click "Choose an existing page to display" and select it first.
Once a first wiki web part has been created and populated with a wiki, any user with "Editor" access to the folder can edit and create additional wikis. See how in the Wiki User Guide .A walkthrough can also be found in this topic: Create a Wiki.
Wiki Web Parts
There are three kinds of wiki web part, two for individual pages and one for the collection as a whole:
The "wide" web part displays one wiki page in the main panel of the page. Add a "Wiki" web part at the bottom of the left side of the page.
The "narrow" web part displays one wiki page on the right side. Add a "Wiki" web part at the bottom of the right column of the page.
The "Wiki Table of Contents" web part can be added on the right side of the page and displays links to all the wiki pages in the folder.
Note that when the browser window is not wide enough to show both side by side, the "narrow" web parts will be shown below the "wide" web parts.
Customizing the Wiki Web Part
To specify the page to display in the Wiki web part, select Customize from the (triangle) menu.
Folder containing the page to display: Select the appropriate container. When you select a folder here, the second dropdown will be populated with the pages available.
Page to display: Select the page name from among those available in the folder chosen. When you click Submit the web part will show the chosen content.
You can use this feature to make wiki content available in folders with different permissions, or even site wide for use in any project. Any wiki page placed in the Shared project can be found by selecting "/Shared" from the folder dropdown.
Customizing the Wiki Table of Contents
The Wiki Table of Contents web part is available on the right hand side of a page. By default, it is named Pages and shows the wikis in the current container. Use the web part > Customize option to change the title or display wikis from another container.
Print Wiki Pages
It can be useful to obtain the "printable" version of a wiki. From the web part menu or editing interface, choose Print to see the printable version in a new browser tab.Choose Print Branch to concatenate any child pages of this page into a single printable new browser tab. This can be especially useful when trying to find a specific section in a nested set of documentation.
Copy Wiki Pages
The (triangle) menu in the header of the Wiki Table of Contents web part (named "Pages") contains these options:
New: Create a new wiki page. Remember to select the page order and parent as appropriate.
Copy: Administrator only. Copy the entire wiki (all wiki pages in the container) to another container. You can either copy only the latest version of each page, or check the box to "Copy Histories Also". See Copy Wiki Pages for additional information.
Print All: Print all wiki pages in the current folder. Note that all pages are concatenated into one continuous document.
This topic describes options available for managing wiki pages. When you are viewing a page, the title is bolded on the Pages list. As you add pages, links are added to the menu creating a table of contents. You can rearrange their order, and can also arrange your table of contents hierarchically, making some pages "parents", with "children" pages underneath.From the wiki web part or wiki page view of the page, select Manage.
The page Name is generally a single short nickname for the page. It must be unique in the container. The Title can be longer and more descriptive, and will be displayed at the top of the page. Note that the page name and title are good places to put any search terms you want to find this page.
Rename a Page
Once a page name is set, you cannot edit it from the editor and must use Manage. Click Rename to change the page name.In order to avoid breaking existing links to the original name, you have the option to add a new page alias for the old name.
Use Page Aliases
When page names change, such as when projects evolve, it can be useful to rename pages but convenient to maintain existing links by using Aliases. When you rename a page you can check the box to add an alias, as shown above, or you can leave the page name as it is and add as many page aliases as you require. Note that if you add an alias that is already the name of another page, the named page will prevail.Once aliases are set, accessing the page via one of those aliases will redirect to the actual page name. Try it by accessing this URL, which is an alias for the page you're reading now. Notice the URL includes the page name, but "manageWikiAliases" will be replaced with the actual name, "manageWiki"
By selecting a Parent page, you can arrange wiki pages into a hierarchy grouping common topics and making it easier for users to browse the table of contents for the wiki as a whole. For example, the parent page of this topic is Wiki Admin Guide. You can see the parent page in the 'breadcrumb' trail at the top of the wiki page view.
Notice sibling pages can be reordered with the Move Up/Move Down buttons. Move "Project X" down and notice the Pages order changes in the panel on the left.
Using the Parent dropdown, select "Project Z (projectz)". This makes Project Z the parent topic for Project X. Notice the "Project X" topic no longer shows any siblings (as "Project Z" has no other child pages).
Click Save.
Notice the Pages web part: any parent page now has a or for expanding and collapsing the table of contents.
For a more elaborate example, browse the LabKey documentation table of contents on the right of the page you are reading now.
LabKey supports copying all of the wiki documentation from one project or folder to another. This can be useful when setting up a new similar project or in a case when you test out a new documentation package before 'deploying' to a public location. You must have administrative privileges on the source folder to copy wikis.
To copy all pages in a folder, follow these steps:
Create the destination folder, if it does not already exist.
In the source folder, create a Wiki TOC web part (it may be named "Pages" by default), if one does not already exist.
Select Copy from the triangle pulldown menu on the Wiki TOC web part.
Select the destination folder from the tree. Note that the source folder itself is selected by default, so be sure to click a new folder if you want to avoid creating duplicates of all pages in the source folder itself.
Check the box provided if you want to Copy Histories Also. Otherwise, all past edit history will only remain with the source wiki.
If a page with the same name already exists in the destination wiki, the page will be given a new name in the destination folder (e.g., page becomes page1).
Copy a Single Page
To copy a single page, make a new empty wiki page in the new location and copy the contents of the page you want to copy. Choosing the same page syntax will make this process easier. Be sure to re-upload any images or other attachments to the new page and be sure to check links after the copy.
A wiki is a collection of documents that multiple users can edit. Wiki pages can include static text and image content, as well as embed live data and reports using queries, visualizations, scripts, etc. Wiki pages can be written using plain text, Wiki syntax, HTML, or Markdown.Once the first wiki web part has been created by an administrator:
Users with update permissions ("Editor" or higher) can perform all the actions described in this topic.
Users with insert but not update permissions ("Submitter/Author") can create new pages.
Users with the site role "Platform Developer" can author and edit pages that incorporate JavaScript <script>, <style>, <object> and similar tags or embedded JavaScript in attributes like onclick handlers.
Learn about how administrators create the first wiki, display wikis in one or more web parts, and manage other aspects in this topic: Wiki Admin Guide.
When viewing a wiki web part, the menu is accessed using the (triangle) menu in the upper right.When viewing a wiki page directly, such as after you click on the title of the web part, you will see links in the header.
History: View past revisions of this page, compare them, and revert if necessary.
Print: Render the wiki in the browser as a printable page. This view can also be generated by appending "_print=1" to the page URL.
Print Branch: This option is shown only if the page has "child" pages, and will print the parent and all children (in display order) in one printable browser window.
Edit a Wiki Page
To edit the wiki shown in a web part, click the pencil icon next to the title or choose Edit from the wiki menu.
Name: Required. The page name must be unique within the folder.
You cannot edit the page name from this interface; use the Manage option to change the name and optionally include a page alias so that links to the original name will continue to work.
Title: The page Title is shown in the title bar above the wiki page, and web part frame, if shown.
Index: Uncheck if the content on this page should not be indexed. Note that it will not be hidden, it will just not show up on searches from other containers.
Parent: The page under which your page should be categorized, if any, creating a hierarchy of wiki 'nodes'.
Using page parents can help users find related topics more easily; the 'ancestors' of a given page are shown in a breadcrumb trail at the top of the page view.
Pages without parents are all shown at the top level of the table of contents.
Body: This section contains the main text of your new wiki page. The sidebar specifies the format in use; this can be changed using the Convert to... button in the upper right.
The buttons above the editor panel are largely self-explanatory:
Save & Close: Saves the current content, closes the editor and renders the edited page.
Save: Saves the content of the editor, but does not close the editor.
Cancel: Cancels out of the editor and does not save changes. You return to the state of the page before you entered the editor.
Delete Page: Delete the page you are editing. You must confirm the deletion in a pop-up window before it is finalized. If the page has any children, you can check the box "Delete an Entire Tree/Node of Wiki Pages" to delete all the child pages simultaneously.
Convert To...: Change the syntax used for this page.
Show/Hide Page Tree: Toggle the visibility of the table of contents within the editor, which can be helpful when linking wikis. This has no effect on the visibility of the table of contents outside of the editor. Use the / icons to expand and collapse nodes in the page tree.
Changing Wiki Format Types
The Convert To... button opens a pulldown menu which allows you to change how the wiki page is defined. Each format option shows a few notes at the bottom of the page about formatting and links.Options from simplest to most complex:
Plain text: Simply enter the text you want to display.
Wiki page: Use special wiki markup syntax to format the text you want to display. This is the recommended format as it is easy to use and contains many powerful formatting features. See also: Wiki Syntax.
Markdown: Enter text using Markdown formatting syntax to control how it is displayed. The markdown gets translated on the fly into HTML. See also: Markdown Syntax.
HTML: Enter HTML markup on the "Source" tab in the body of the wiki editor to format your text and page layout.
Note that when using the "HTML" syntax option, there is a "simple" page editor available from the menu and you'll need to click Advanced Editor to open the full editor.
The "Visual" input tab allows you to define and layout the page using common document editor tools and menus. Additional control and options are also possible when directly entering/editing the HTML source for the page.
Menus available include "Edit, Insert, View, Table, and Help". Toolbar buttons let you undo, redo, set the style (Div > Headings, Inline, Blocks, Align), font size, as well as bold, italic, underline, strikethrough, and more. Hover over any tool to learn more. Click the ellipsis to show any remaining tools on narrower browsers.
When a page contains elements that are not supported by the visual editor, you will be warned before switching from the "Source" to "Visual" tab that such elements would be removed if you did so. Examples include but are not limited to <i>, <form>, <script>, and <style> tags.
Best practice is to save changes on the "Source" tab before switching to the "Visual" tab, and vice versa. If you want to check how the wiki will appear, do so using a separate browser tab. Not all elements on the visual tab will exactly match the end view. You may also want to copy the HTML content to an outside document for safekeeping and comparison if you are concerned.
If you accidentally save a document that is missing any content removed by switching to the "Visual" tab, you can restore the prior version from the page History.
Please note that your content is not automatically converted when you switch between rendering methods. It is usually wise to copy your content elsewhere as a backup before switching between rendering modes, particularly when using advanced layouts. You may also choose to open a view of a wiki page in one browser window and the editor in another, particularly when changing from an HTML-source page to a simpler to edit type like Wiki syntax.
Using File Attachments in Wikis
Below the body in the wiki editor, the Files section enables attaching images, downloadable files, and other content to your page.
Add Files: Click "Attach a file", then "Browse or Choose File" to locate the file you wish to attach. Within the popup, select the file and click "Open." The file will be attached when you save the page. Two files uploaded to the same page cannot have the same name. To replace an attachment, delete the old attachment before adding a new one of the same name.
Delete Files: Click Delete next to any attached file to delete it from the page.
Show Attached Files: If you check this box, the names of the files are rendered at the bottom of the displayed page. This can be useful when attachments should be downloaded, but unnecessary when the attachments are images displayed within the text.
If you want the user to be able to download a non-image file, without showing the attached filenames at the bottom, in wiki syntax you can use the same square bracket linking syntax as for images:
[FILENAME] OR [Display text|FILENAME]
Show/Hide Attachment List
The list of files and images can be shown at the bottom of a wiki page, or hidden by using the Show attached files checkbox within the editor. When attachments are shown, they will be listed at the bottom with a divider like this:If you are writing an HTML format wiki and wish to hide the "Attached Files" line border, include this styling in your page source:
To add images to a wiki-language page, first add the image as an attachment, then refer to it in the body of the wiki page:
Wiki syntax: [FILENAME.PNG].
HTML source editor: <img src="FILENAME.PNG"/>
HTML visual editor: Place the cursor where you want the image, and either click the image icon on the toolbar, or select Insert > Image. Enter the filename as the Source and click Save. Note that the visual editor panel does not display your image, but you can see "img" in the Path shown at the bottom of the panel when your cursor is on the location of the image.
Embed Data or Other Content in a Wiki
You can embed other "web parts" into any HTML wiki page to display live data or the content of other wiki pages. You can also embed client dependencies (references to JavaScript or CSS files) in HTML wiki pages with sufficient permissions.Learn more about embedding content in this topic: Embed Live Content in HTML Pages
Only admins and users with the "Platform Developer" role can create or edit wiki pages that use JavaScript <script> tags or embedded JavaScript in attributes like onclick handlers.
Manage a Wiki Page
Click Manage to configure elements of the wiki page other than the contents, such as name, parent, and alternate name aliases.Learn more in this topic:
When you delete a wiki page that is the parent of one or more other pages, you have the option of simultaneously deleting all the children (and subchildren if such exist). From the parent page, click Manage or Edit, then Delete. Check the box to "Delete Entire Wiki Subtree."
Wiki Table of Contents
The web part Wiki Table of Contents lists all pages in the folder to help you navigate among them. The page you are viewing will appear in bold italics (for this page, Wiki User Guide). Click on any page to open it.
/ icons: Expand or collapse parent/child nodes within the wiki.
Expand/Collapse All: Use these links to expand or collapse all nodes in the wiki.
Previous/Next: Use these links to page through the wiki.
The (triangle) menu in the header of the Pages wiki table of contents web part contains these options:
New: Create a new wiki page. Remember to select the page order and parent as appropriate.
Copy: Administrator only. Copy the entire wiki (all wiki pages) to another container. See also : Wiki Admin Guide
Print All: Print all wiki pages in the current folder. Note that all pages are concatenated into one continuous document.
When creating a page using the format "Wiki Page", you use wiki syntax to format the page. The following table shows commonly used wiki syntax.
See the Wiki Syntax: Macros page for further options.
Syntax
Effect
**bold text**
bold text
__underline text__
underline text
~~italicize text~~
italicize text
1 Main Heading
Main Heading
1.1 Sub Heading
Sub Heading
1.1.1 Minor Heading
Minor Heading
1.1.1.1 Inner Heading
Inner Heading
- Top Level Bullet -- Second Level Bullet --- Third Level Bullet
In documents written in wiki syntax, macros can make it easy to get the look and behavior you need. This topic lists the available macros and provides several usage examples.
The following macros work when encased in curly braces. Parameters values follow the macro name, separated by colons. Escape special characters such as "[" and "{" within your macro text using a backslash. For example, {list-of-macros} was used to create the following table:
Macro000000
Description
Parameters
anchor
Anchor Tag
name: The target anchor name. Example: {anchor:name=section1}
code
Displays a chunk of code with syntax highlighting, for example Java, XML and SQL. The none type will do nothing and is useful for unknown code types.
1: syntax highlighter to use, defaults to java (optional)
comment
Wraps comment text (which will not appear on the rendered wiki page).
none
div
Wraps content in a div tag with an optional CSS class and/or style specified.
class: the CSS class that should be applied to this tag. style: the CSS style that should be applied to this tag.
Example: {div:class=bluebox}...{div}
file-path
Displays a file system path. The file path should use slashes. Defaults to windows.
1: file path
h1
Wraps content in a h1 tag with an optional CSS class and/or style specified.
class: the CSS class that should be applied to this tag. style: the CSS style that should be applied to this tag.
image
Displays an image file.
img: the path to the image. alt: alt text (optional) align: alignment of the image (left, right, flow-left, flow-right) (optional)
Base LabKey macro, used for including data from the LabKey Server portal into wikis.
tree : renders a LabKey navigation menu. treeId: the id of the menu to render can be one of the following: core.projects, core.CurrentProject, core.projectAdmin, core.folderAdmin, core.SiteAdmin
The following wiki code...{span:style=text-decoration:line-through}
Strike through this text.
{span}...renders text with a line through it.
Strike through this text.
Example: Using the {div} Macro to Apply Inline CSS Styles
The {div} macro lets you inject arbitrary CSS styles into your wiki page, either as an inline CSS style, or as a class in a separate CSS file.The following example demonstrates injecting inline CSS styles to the wiki code.The following wiki code...{div:style=background-color: #FCAE76;
border:1px solid #FE7D1F;
padding-left:20px; padding-right:15px;
margin-left:25px; margin-right:25px}
- list item 1
- list item 2
- list item 3
{div}...renders in the browser as shown below:
list item 1
list item 2
list item 3
Example: Using the {div} Macro to Apply CSS Classes
To apply a CSS class in wiki code, first create a CSS file that contains your class:
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
The anchors created will be the heading text, all lower case and substituting "-" for spaces. For example, this heading:
### My Section Heading
Would be referenced in a markdown link as:
[on page link text](#my-section-heading)
[cross page link text](./wiki-page.view?name=anotherPage#my-section-heading)
Special Wiki Pages
Wiki pages that begin with an underscore have special behavior, including two special case wikis that use specific filenames.
Underscores in Wiki Names
Wikis whose names start with an underscore (for example: "_hiddenPage") do not appear in the wiki table of contents, for non-admin users. For admin users, the pages are visible in the table of contents.
_termsOfUse
A wiki page named "_termsOfUse" will require users to agree to a terms of use statement as part of the login process. For details, see Establish Terms of Use.
_header
A wiki page named "_header" will replace the default "LabKey Server" logo and header. Adding the _header wiki to a project means it will be applied to all of the folders within that project.You can also change the logo by adjusting "Look and Feel" settings at the project- or site-level.
You can embed live content in wiki pages by inserting web parts or dependencies using substitution syntax. For example, you might embed a live table of currently open issues on a status page, or include a key visualization to illustrate a point in the description of your research protocol.
Embed a web part or visualization in an HTML wiki page.
Combine static and dynamic content in a single page. This eliminates the need to write custom modules even when complex layout is required.
Embed wiki page content in other wiki pages to avoid duplication of content (and thus maintenance of duplicate content). For example, if a table needs to appear in several wiki pages, you can create the table on a separate page, then embed it in multiple wiki pages.
Indicate JavaScript or CSS files that are required for code or styles on the page; LabKey will ensure that these dependencies are loaded before your code references them.
Embedding web parts and dependencies is available only on pages that use HTML syntax.Security rules are respected for inserted web parts. To see inserted content, a reader must have viewing permissions for both:
The displaying page.
The source container for the content inserted in the page.
Only administrators and platform developers can create or edit HTML pages that include elements with tags <link>, <style>, <script>, <script object>, <applet>, <form>, <input>, <button>, <frame>, <frameset>, <iframe>, <embed>, <plaintext>, or embedded JavaScript in attributes like onclick handlers.
Embed Web Parts
To embed a web part in an HTML wiki page, open the page for editing and go to the HTML Source tab. (Do not try to preview using the Visual tab, because elements may be removed.) Use the following syntax, substituting appropriate values for the substitution parameters in single quotes:
For example, if you wanted to have a wiki with a bit of text, then a query, followed by some more text, you could nest the web part in other HTML. In this example, we'll show a list without the query web part framing:
<p>This is some text about the query shown here, the Languages list in this folder:</p>
You can find the web part names to use as the 'partName' argument in the Web Part Inventory. These names also appear in the UI in the Select Web Part drop-down menu.
Configuration Properties for Web Parts
The Web Part Configuration Properties page covers the configuration properties that can be set for various types of web parts inserted into a web page using the syntax described above
Examples
You can find a page of examples, showing web parts of a number of types embedded in the topic: Examples: Embedded Web Parts. That page includes a view source link to the HTML source for its examples.Report
For a wiki page in a different container, use the webPartContainer property. To get the webPartContainer for the source container, see Web Part Configuration Properties.
<div id="targetDiv"></div> <script> var wp = new LABKEY.WebPart({ renderTo: 'targetDiv', partName: 'List - Single', partConfig: { listName: 'MyList'}}); wp.render(); </script>
Assay Runs Example:For assay runs, you need to use viewProtocolId, equivalent to the RowId of the assay design. It's in the URL for most assay pages.
<div id="targetDiv2"></div> <script> var wp2 = new LABKEY.WebPart({ renderTo: 'targetDiv2', partName: 'Assay Runs', partConfig: { viewProtocolId: 26027}}); wp2.render(); </script>
Search Example:The following snippet embeds a Search web part in a wiki page. Setting location='right' means that the narrow wiki part is used, so the web part will display in the right column of a page.
This wiki page contains examples of web parts embedded in wiki pages. Review the permissions requirements and more detailed explanations of this feature in this topic: Embed Live Content in HTML Pages.
Click view source to review the syntax that inserts each of these web parts.
This tutorial walks you through capturing, importing, and interpreting a spreadsheet of experimental results. In particular, this tutorial will show how to:
Capture the experiment data in LabKey Server using a new Assay Design.
Import the data to the server.
Visualize the data.
As part of this tutorial, we will create an Assay Design, a data structure for the import and storage of experiment data. The Assay Design is based on the General assay type, the most flexible of LabKey Server's tools for working with experiment data. This assay type gives you complete control over the format for mapping your experimental results into LabKey Server.
The tutorial provides example data in the form of a Cell Culture assay that compares the performance of different media recipes.
Web parts can be customized using a variety of properties, detailed in this topic. These properties can be used when displaying a web part in a wiki using the embedding syntax. Not all of these properties and options are available via the user interface, and those that are sometimes have different naming there, or are only available in page admin mode.
Subfolders: A preconfigured projects web part for subfolders.
Properties Common to All Web Parts
Two properties exist for all web parts. These properties can be set in addition to the web-part-specific properties listed below.
showFrame: Whether or not the title bar for the web part is displayed. For example, for wiki pages, the title bar includes links such as "Edit" and "Manage". You will want to set showFrame='false' to show web part content seamlessly within a wiki page, for example. Values:
true (the default)
false
location: Indicates whether the narrow or wide version of the web part should be used, if both options are available. Including a second location setting of 'menu' will make the web part available for use as a custom menu.
content (the default)
right (aka 'narrow')
menu
Not all web parts provide different formatting when the location parameter is set to 'right'. For example, the Wiki web part does not change its display, it merely tightens the word wrap. Others (such as Protein Search, Sample Types, Files) change their layout, the amount of data they display, and in some cases the functions available.
Properties Specific to Particular Web Parts
Properties specific to particular web parts are listed in this section, followed by acceptable values for each. Unless otherwise indicated, listed properties are optional.
Assay Runs
viewProtocolId: The rowID (sometimes referred to as the protocolID) for the assay design. You can find it in the URL for most assay pages.
Files
The Files web part supports uploading and browsing of files in the container.
fileSet: When one or more named file sets are defined, use this parameter to select which to display. Learn more here: Named File Sets.
folderTreeVisible: Whether to show the folder tree panel on the left.
fileRoot: Set to use a subfolder as the root for this Files web part. Include the "@files" root in the value. For example: fileRoot : '@files/subfolder/'
Issues List/Summary
The Issues List and Issues Summary web parts list issues in the current folder's issue tracker.
title: Title of the web part. Useful only if showFrame is true. Default: "Issues List" or "Issues Summary" respectively.
List - Single
listName: The name of the list.
Query
The Query web part shows the results of a query as a grid.
allowChooseQuery: Boolean. If the button bar is showing, whether to let the user choose a different query.
allowChooseView: Boolean. If the button bar is showing, whether to let the user choose a different view.
buttonBarPosition: String. Determines whether the button bar is displayed.
none
top
queryName: String. The query to show. If omitted, then the list of queries in the schema is shown.
schemaName: (Required) String. Name of schema that this query is defined in.
shadeAlternatingRows: Boolean. Whether the rows should be shaded alternating grey/white.
showDetailsColumn: Boolean. Whether to show the detail links for each row.
showFrame: Boolean. If false, the title is not shown.
showUpdateColumn: Boolean. Whether to show the selecter column for each row.
title: Title to use on the web part. If omitted, the query name is displayed.
viewName: Custom view to show from that query or table.
reportId: The ID of the report you wish to display, shown in the URL. If the URL includes 'db:151', the reportID would be 151.
schemaName, queryName and reportName: You can use these three properties together as an alternative to reportId. This is a handy alternative in scripts designed to run on both test and production systems when the reportID would be different.
showSection: Optional. The section name of the R report you wish to display. Optional. Section names are the names given to the replacement parameters in the source script. For example, in the replacement '${imgout:image1}' the section name is 'image1'. If a section name is specified, then only the specified section will be displayed. If not, by default, all sections will be rendered.
The Search web part is a text box for searching for text strings.
includeSubFolders: Whether to include subfolders in the search
true (the default)
false
Wiki
name: Required. Title name of the page to include.
webPartContainer: The ID of the container where the wiki pages live. Default: the current container. Obtain a container's ID either by going to > Folder > Management and clicking the Information tab, or by replacing the part of the URL after the last slash with "admin-folderInformation.view?". For example, to find the container ID of the home project on a local dev machine, use:
title: Title for the web part. Only relevant if showFrame is TRUE. Default: "Pages".
webPartContainer: The ID of the container where the wiki pages live. Default: the current container. Obtain a container's ID as described above.
Projects
The Projects web part displays the projects or folders in a variety of ways that can be customized. When you use the user interface, the names of selectable options differ somewhat from the parameters used internally, listed here.
containerTypes: Which content to display in this web part:
project (the default)
folder
iconSize: What size icons to show
large (the default)
medium
small
labelPosition: Where to display the text label. You will want to use 'side' if the iconSize is small.
bottom (the default)
side
containerFilter: Which containers to show for the containerType set. Typical combinations are Project/CurrentAndSiblings and Folder/CurrentAndFirstChildren.
CurrentAndSiblings
CurrentAndFirstChildren
includeWorkbooks: Whether to include workbooks as "subfolders"
true
false
hideCreateButton: Whether to hide the "Create New Project" button.
false (the default - i.e. by default the button is shown)
true
Subfolders
The subfolders web part can be added to a page separately through the user interface. It has the same configuration properties as a Projects web part, with the following settings preapplied:
Screen captures or other images can be added to wikis to illustrate your text. For example, many LabKey documentation pages use screen captures to show steps users need to follow on support or help pages. This topic covers how to add these images to a wiki document.
The following screen capture shows paint.net in action and includes circles around key features discussed below.
Capture and Open a Screen Image
Methods for taking a screen capture of your desktop vary by platform. Look for a print screen key or "Snipping Tool" utility.Capture the entire screen or a subset showing the area of interest. It may also help to resize your browser display window prior to capturing. In the LabKey Documentation, we aim for 800px wide images for a good display width.Save the capture, typically as a PNG, jpg, or similar file. It may give you an option to choose the save location. Open the image in paint.net.
Crop and/or Resize the Image
The "Tools" panel typically floats on the left edge of the paint.net canvas. If this, or any tool, disappears, you can click the boxes in the upper right to show them again.
In the "Tools" panel, select the "Rectangle Select," represented by a dashed rectangle.
Click and drag the selection rectangle across the region of interest in your desktop screenshot.
Click the now-activated crop icon in the menu bar.
You may wish to repeat this process a few times to refine the selected region. If necessary, use the 'back' action in the history panel to undo any action.
To resize a large image:
Use the "Image" drop-down to select "Resize."
Make sure that "Maintain aspect ratio" is checked to allow the image to be shrunk in a uniform way in all directions.
Choose a new width for the image. LabKey documentation wikis typically keep images smaller than 800 pixels wide; given the border added below, this means a max-width of 798 pixels.
Annotate the Image
Using outlining, arrows, and text can help call out interesting features. First select the color palette you want to use. For LabKey documentation we use an orange (use hex E86030) color, you can also use red or any color you like.
In the "Colors" panel, you can click the wheel, or use the More >> menu to select colors by hex code, RGB, HSV, etc.
The 'Primary' color will be used for outlines, arrows, and text.
The 'Secondary' color will be added to the background if you expand the image (to add an outline below) or if you fill a shape).
In the "Tools" panel, select the "Shape" button at the bottom of the menu, then choose "Rounded Rectangle" from the pulldown in the menu bar.
Click on your image and drag the rounded rectangle across the image to add a highlight outline to the image.
In the "Tools" panel, click the 'T' button to enter text.
The font and size will be selectable above the workspace.
Type into the image as you feel is necessary and helpful.
After you add text you will see a handle for moving it around.
Arrows can also be added using the line icon with a "Style" selection including an arrow at the end.
Add a Pointer or Cursor Image
Download one of the cursor images:
Select "Layers" -> "Import from File" and browse to the downloaded cursor image.
Position the cursor image as appropriate using the handle.
After adding a cursor, note that the file will be in several layers (shown in the "Layers" panel). When you save, you will want to specify saving as a PNG file and accept flattening of the image. By default, multilayer files will be saved as workspaces and not display correctly until flattened.
Add a Border
Use the "Image" drop-down to select "Canvas Size."
Make sure that the image icon in the "Anchor" section of the popup is centered so that the canvas will be expanded in equal amounts on all sides.
Set the fill color to Black or the desired outline.
Increase "Width" and "Height" in the "Pixel size" section by 2 pixels each. This adds a one-pixel border on each side of the image.
Save the Image
Typically, save the image as a .png for use on a web site.
You will be prompted to flatten the image if you imported cursors or other image files.
Add your Image to a Wiki-syntax Page
Open a wiki page for editing.
Add the saved file as an attachment to the wiki.
Uncheck the "Show Attached Files" box (unless you also want a list of image names visible to the user).
At the point in the text where you would like to display the image, enter the name of the image file enclosed in square brackets. Example: [myImage.png].
Typically you would place the image on its own line, and it will be left-justified. To add an image 'floating' on the right, enclose it in formatting like this:
{span:style=float:right;margin-top:-10px;margin-left:-30px;}[myImage.png]{span}The text to float to the left....
Save and close the wiki.
Add your Image to an HTML-syntax Page
The following steps outline using the Visual tab for adding an image.
Open a wiki page for editing.
Add the saved file as an attachment to the wiki.
Uncheck the "Show Attached Files" box (unless you also want a list of image names visible to the user).
At the point in the text where you would like to display the image, click the Image icon in the wiki editor, or select Insert > Image.
Enter the image name as the "Source". Optionally, enter an "Alternative description". You can adjust the width and height here, but it is recommended that you do so in the image itself. Click Save.
You may not see the image itself in the editor at this point.
Save and close the wiki.
The image will appear where you placed it.
If you are using the Source tab:
Open a wiki page for editing.
Add the image file as an attachment to the wiki.
Uncheck the "Show Attached Files" box (unless you also want a list of image names visible to the user).
Use syntax like the following where you want to show the image:
The LabKey Issues module provides an issue tracker, a centralized workflow system for tracking issues or tasks across the lifespan of a project. Users can use the issue tracker to assign tasks to themselves or others, and follow the task through the work process from start to completion.
Note: All issue trackers on your LabKey Server site are stored in the same underlying database table, and issue numbers are assigned sequentially as issues are added, regardless of the project or folder to which they belong. Issue numbers in any given project or folder list may not be contiguous, though will be in sequence.
The issue tracking tool helps you manage complex tasks, especially tasks that have multiple steps and require input from multiple team members. It is also useful for keeping track of small tasks that can easily fall through the cracks of a busy distributed team.
Issues move through phases, using different status settings.
The open phase - The issue has been identified, but a solution has not yet been applied.
The resolved phase - A solution has been applied or proposed, but it is not yet confirmed.
The closed phase - The solution is confirmed, perhaps by the person who identified the problem, and so the issue is put to rest.
There are three main areas: a title section (with action links), a properties/status report area, and series of comments. Each comment includes the date, the contributing user, life-cycle status updates, and the body of the comment, which can include file attachments.
Set Up Tutorial
To complete this tutorial, you can use any folder where you have administrator access and it is suitable to do testing. For example:
Log in to your server and navigate to your "Tutorials" project, "Collaboration Tutorial" subfolder. Create them if necessary, accepting default types and settings.
If you don't already have a server to work on where you can create projects and folders, start here.
If you don't know how to create projects and folders, review this topic.
Create an Issue Tracker
First we add the issue tracker to the folder. There can be several different types of issue trackers, and issue definitions are used to define the fields and settings. In this case we need only a general one that uses default properties.
Navigate to the Collaboration Tutorial folder, or whereever you want to create the issue tracker.
If you do not see this option on the list of web parts, you may need to enable the issues module in your folder. Select > Folder > Management > Folder Type, check the box for the Issues module, and click Update Folder.
Click (Insert New Row) in the new web part.
Give the new definition a Label. For the tutorial, use "My New Tracker".
Select the Kind: "General Issue Tracker" is the default.
Click Submit. Confirm that you are creating a new definition in your local folder.
If there is another issue tracker with the same name in the same folder or "Shared" project (searched in that order), your new tracker will share the same definition. You'll be asked to confirm before the tracker is created.
Note: If you are sharing a definition, particularly with one in the "Shared" folder, be aware that changes you make here will change the definition for all issue trackers that use it.
Review the properties and default fields, but make no changes.
Notice that the setting for Populate 'Assigned To' Field from is "All Project Users". This means all users who have the role of "Editor" or higher and are members of at least one project security group.
Confirm that your new default definition is selected.
Providing a title is optional. For the tutorial, you could use "Tutorial Issues".
Click Submit.
The page now has a new web part containing an empty issue list. The web part will have the title you gave it, or a default based on the issue list definition you used.
You can now delete the "Issue Definitions" web part to declutter the page. Select Remove from Page from the menu in the web part.
Note that removing a web part merely removes the UI panel. It does not remove the underlying content.
Click Exit Admin Mode in the upper right.
Grant Roles to Users
To be able to create issues, we need to have users to whom they can be assigned. The default setting we chose for the issue tracker includes all users who have the role of "Editor" or higher and are members of at least one project security group. When you first create a project, even you the site admin are not a member of a project group so cannot be assigned issues.In this step we use the project group "Users" but you could also create your own group or groups of users; as long as they all have "Editor" access or better, they can be assigned issues under this setting.
Select > Folder > Permissions.
Click the Project Groups tab.
If you see "Users" listed, click it. Otherwise, type "Users" in the New Group Name box and click Create New Group.
In the Add user or group... dropdown, select yourself. You can add other users as well if you like.
Click Done in the pop-up.
Click Save and Finish.
Next we elevate the "Users" group to the Editor role.
Select > Folder > Permissions.
If necessary, uncheck Inherit permissions from parent.
Using the dropdown next to the Editor role, select "Users". This means that anyone in that group can open and edit an issue. Clicking the group name will open a popup listing the members.
Click Save.
Click the tab Project Groups.
Click Users. Notice that Editor is now listed under Effective Roles.
Click Done.
Click Save and Finish.
Use the Issue Tracker
Now you can open a new issue:
In the Issues List web part, click the New Issue button.
On the Insert New Issue page:
In the Title field: "Some Task That Needs to be Completed"
From the Assign To dropdown, select yourself.
In the Comment field, enter "Please complete the task." Notice below the field that you could update a supporting file attachment if needed.
Notice that the other required field (marked with an asterisk), Priority, has a default value of 3 already. Defaults are configured on the issues admin page.
Click Save.
You will see the details page for your new issue.
Click the Collaboration Tutorial link to return to the main tutorial page.
Notice that the new issue appears in the Issues List web part and you can click the title to return to the details. From the detailed view, you can also click links to:
Update: Add additional information without changing the issue status. For example, you could assign it to someone else using an update.
Resolve: Mark the issue resolved and assign to someone else, possibly but not necessarily the person who originally opened it, to confirm.
Close(available for resolved issues): Confirm the resolution of the issue closes the matter.
Reopen(available for resolved issues)): Reopen the issue, generally with a comment clarifying why it is not resolved.
More: Pull down a few more options:
New Issue
Create Related Issue: The newly created issue will have this issue's number in the "Related" field.
Email Preferences: Set when and how you would like to receive email notifications about this and other issues.
Print: Generate a printable view of the issue, including the comment history.
If you've added other users (real or fictitious ones as part of another tutorial) you could now assign your practice issue to one of them, then impersonate them to simulate a multi-user issue tracking process. When finished experimenting with your issue, you can close it:
Open the detail view of the issue by clicking the title in the issue list.
Click Resolve.
By default the Assigned to field will switch to the user who originally opened the issue (you); when there are more users in the group, you may change it if needed, so that a different person can confirm the fix.
Notice the options available on the Resolution field. The default is "Fixed" but you could select a different resolution, such as "By Design" for something that does not need fixing but might need explaining. For the tutorial, leave the default "Fixed" resolution.
Enter a message like "I fixed it!" and click Save.
You could return to the issue list to see how it looks now, then reopen it, or stay in the editor.
Now instead of a Resolve button, there is a Close button. Click it.
Enter a message if you like ("Well done.") and click Save.
Your issue is now closed.
See a working example of a similar issue list: Issues List.
An Issue Tracker can be used to track any tasks or work that needs to be done by a group of users. Once added by an administrator, it usually appears to the user in an Issues List web part. The issue grid is like any other data grid offering options to sort, filter, customize, export, and chart the data. In addition an issue tracking workflow is preconfigured.
An issue has one of three possible phases: it may be open, resolved, or closed. Between the phases, updates can add information and the assignment can change from user to user.To get started using issue trackers, review this topic:
When you open an issue, you provide the title and any other necessary information, and a unique issue ID is automatically generated. You also must assign the issue to someone from the Assigned To pulldown list, which is configured by an administrator. Other site users, i.e. stakeholders, may be added to the Notify List now or when the issue is updated later.The person to whom the issue is assigned and all those on the notify list will receive notification any time the issue is updated.
Update an Issue
When an issue is assigned to you, you might be able to resolve it in some manner right away. If not, you can update it with additional information, and optionally assign it to someone else if you need them to do something before you can continue.For example, if you needed test data to resolve an issue, you could add a request to the description, reassign the issue to someone who has that data, and add yourself to the notify list if you wanted to track the issue while it was not assigned directly to you.You can update an open or a resolved issue. Updating an issue does not change its status.
Create a Related Issue
As you proceed to work through resolving an issue, it may become clear that there is a secondary or related issue. Tracking such issues independently can help with prioritization and clear resolution to all parts of an issue.To create a related issue, select Create Related Issue from the More > menu while viewing any issue. You'll select the container in which to open the related issue, and the current issue will be prepopulated as "Related" to the new one.An administrator can configure the default folder in which the related issue will be created, supporting a workflow where related tickets are used to 'escalate' or generalize an issue from many separate projects to a common location for a different group to monitor. You can change the folder from the dropdown if the default is not what you intend.
Resolve an Issue
Declaring an issue resolved will automatically reassign it back to the user who opened it who will decide whether to reopen, reassign, further update, or ultimately close the issue.Options for resolution include: Fixed, Won't Fix, By Design, Duplicate, or Not Repro (meaning that the problem can't be reproduced by the person investigating it). An administrator may add additional resolution options as appropriate.When you resolve an issue as a Duplicate, you provide the ID of the other issue and a comment is automatically entered in both issue descriptions.
Close an Issue
When a resolved issue is assigned back to you, if you can verify that the resolution is satisfactory, you then close the issue. Closed issues remain in the Issues module, but they are no longer assigned to any individual.
A note about filtering: When you filter the default view of the Issues List grid, your filters also apply to fields within the details view of individual issues. For example, if you filter out "Closed" issues, you will no longer see any closed issue IDs in the "Related Issues" section.
Some data grid features that may be particularly helpful in issue management include:
View Selected Details
To view the details pages for two or more issues, select the desired issues in the grid and click View Selected Details. This option is useful for comparing two or more related or duplicate issues on the same screen.
Specify Email Preferences
Click Email Preferences to specify how you prefer to receive workflow email from the issue tracker. You can elect to receive no email, or you can select one or more of the following options:
Send me email when an issue is opened and assigned to me.
Send me email when an issue that's assigned to me is modified.
Send me email when an issue I opened is modified.
Send me email when any issue is created or modified.
Send me email notifications when I create or modify an issue.
Click Update to save any changes.
Issues Summary
The Issues Summary web part can be added by an administrator and displays a summary of the "Open" and "Resolved" issues by user.
Click any user name to see the issues assigned to that user.
Click the View Open Issues link to navigate to the full list of open issues.
Not Supported: Deleting Issues
LabKey Server does not support deleting issues through the user interface. Typically, simply closing an issue is sufficient to show that an issue is no longer active.
This topic is under construction for the 25.7 (July 2025) release. For the previous documentation of this feature, click here.
An issue tracker can be configured to suit a wide variety of workflow applications. To define an issue tracker, an administrator first creates an Issue List Definition (the properties and fields you want), then creates an Issues List using it. Multiple issue trackers can be defined in the same folder, and issue list definitions can be shared by multiple issues lists across projects or even site-wide.
First, define an Issue List Definition. Most issue trackers use the "General Issue Tracker" kind. The Label (Name) you give it can be used to share definitions across containers.
When you first create an issues list, you will be able to customize properties and fields. To reopen this page to edit an existing issue tracker's definition, click Admin above the issues grid.The issues admin page begins with Issues List Properties:Define properties and customize Fields before clicking Save. Details are available in the following sections.
Comment Sort Direction
By default, comments on an issue are shown in the order they are added, oldest first. Change the Comment Sort Direction to newest first if you prefer.
Item Naming
The Singular item name and Plural item name fields control the display name for an "issue" across the Issues module. For example, you might instead refer to issues as "Tasks", "Tickets", or "Opportunities" depending on the context.
Assignment Options
You can control the list of users that populates the Assigned To dropdown field within the issue tracker. By default, the setting is "Site: Users", meaning any user logged into the site. You may select "Users" instead, meaning the project level "Users" group, which includes all users who are members of project level groups.
You can also select another site or project group from the dropdown menu.Only users who also have the "Editor" role (or higher) in the folder will be listed on the dropdown. This is required to allow them to report progress by editing any issue they are assigned.
Default User Assignment
You also have the option of selecting the Default User Assignment for new issues. This can be useful when a central person provides triage or load balancing functions. All users in the selected Assigned To group will be included as options in this list.
Default Folder for Related Issues
When a user creates a related issue from an issue in this tracker, you can set the default folder for that issue. This is a useful option when you have many individual project issue trackers and want to be able to create related issues in order to 'escalate' issues to a shared dashboard for developers or others to review only a subset.Default folders can be used with Shared issue list definitions, though apply on a folder by folder basis. Set a folder-specific value for this field along with other folder defaults.
Customize Fields
Click the Fields section to open it. You will see the currently defined fields. General Issue Trackers (the default Kind) include a set of commonly used fields by default, some of which are used by the system and cannot be deleted.You can use the field editor to add additional fields and modify them as needed.
Check the Required box for any fields that you want users to always populate when creating an issue. Use default values when appropriate.
Click the icon on the right to expand field details, including lookup targets and advanced settings.
Use the six-block handle on the left to drag and reorder fields.
Any field showing a on the right is not required by the system and may be deleted if not needed.
Fields that are lookups, shown here in the case of the "Resolution" property, present users with a picklist menu of options. The field details panel shows the list that will be used; here "mynewtracker-resolution-lookup". Expand the field details if you want to change the lookup. Click the lookup target name to go to the list it is looking up from.
Default Values
To configure a field to provide users with a default value, click to expand it, then click Advanced Settings. The current type and value of the default will be shown (if any).The Default Type can be fixed, editable, or based on the last value entered. Select the type of default you want from the dropdown. If you change it, click Apply and then Save; reopen the editor if you want to now set a default value:To set a Default Value for a field, click to expand any field. Click Advanced Settings. Click Set Default Values to open a page for editing all fields with default values enabled may be edited simultaneously:
Hide Protected Fields
To require that a user have insert permissions (Submitter, Editor, or higher) in order to see a given field, select PHI Level > Limited PHI on the Advanced Settings popup. This allows certain fields to be used by the issue tracking team, but not exposed to those who might have read-only access to the issue tracker.Learn more in this topic:
If you want to offer the user a "pick list" of options, you will populate a list with the desired values and add a field with the type Lookup into the appropriate list. Built in fields that use selection options (or pick lists) include:
Type: the type of issue or task
Area: the area or category under which this issue falls
Priority: the importance of this issue
Milestone: the targeted deadline for resolving this issue, such as a release version
Resolution: ways in which an issue can be resolved, such as 'fixed' or 'not reproducible'
The list of options for each field is typically named combining the issue tracker name and field name. When editing the fields in an issue tracker definition, the name of the lookup target will link to the selection list. For example, in an issue tracker named "My New Tracker", the selection list for the "Resolution" field is named Lists.mynewtracker-resolution-lookup:Click to open it. It looks like this (by default) and can be edited to change the available selections:When a user is resolving an issue, the pulldown for the Resolution field will look like this:To add, change, or remove options, select > Manage Lists and edit the appropriate list.
URL Properties
If you would like to have a custom field that links to an outside resource, you can use a URL field property. To apply a URL field property, do not use the issue tracker admin page; instead you must edit the metadata for the issue tracker via the query browser:
Select > Go To Module > Query.
Open the issues schema, then your specific issue list definition.
Save your change and return to see that the values will now link to the target you specified.
Share an Issue List Definition
In some cases, you may want to use the same issue list definition in many places, i.e. have the same set of custom fields and options that you can update in one place and use in many. This enables actions like moving issues from one to another and viewing cross-container issues by using a grid filter like "All folders".Issue list definitions can be shared by placing them in the project (top-level folder), for sharing in all folders/subfolders within that project, or in the /Shared project, for sharing site-wide. Note that an issue list definition in a folder within a project will not be 'inherited' by the subfolders of that folder. Only project level issue list definitions are shared.When you define a new issue list definition in any folder, the name you select is compared first in the current folder, then the containing project, and finally the "Shared" project. If a matching name is found, a dialogue box asks you to confirm whether you wish to share that definition. If no match is found, a new unique definition is created. An important note is that if you create a child folder issue list definition first, then try to create a shared one with the same name, that original child folder will NOT share the definition.Issue list definitions are resolved by name. When you create an issue tracker in a folder, you select the definition to use. If the one you select is defined in the current project or the /Shared project, you will be informed that the shared definition will be used.
Customize the Local Version of a Shared Issue Tracker
When you click the Admin button for an issue tracker that uses a shared issue list definition, you will see a banner explaining the sharing of fields, with a link to manage the shared definition.You can edit Issues List Properties for the local version of this issue tracker, such as the sort direction, item name, and default user assignment. These edits will only apply to your local issue tracker.Note that you cannot edit the Fields or their properties here; they are shown as read only. Click the words source container in the banner message, then click Admin there to edit the fields where they are defined. Changes you make there will apply to all places where this definition is shared.Note that you can edit the default values for field properties in your local container. To do so, open the default value setting page for the shared definition, then manually edit the URL to the same "list-setDefaultValuesList.view" page in your local container. Defaults set there will apply only to the local version of the issue tracker. You can also customize the dropdown options presented to users on the local issue tracker by directly editing the XML metadata for the local issue tracker to redirect the lookup.
Move an Issue to another Issue Tracker
If you organize issues into different folders, such as to divide by client or project, you may want to be able to move them. As long as the two issue lists share the same issue definition, you can move issues between them. Select the issue and click Move. The popup will show a dropdown list of valid destination issue lists.
Customize Notification Emails
Notification emails can be sent whenever an issue is created or edited in this folder. By default, these emails are automatically sent to the user who created the issue, the user the issue is currently assigned to, and all users on the Notify List.Click Customize Email Template in the header of the issues list to edit the template used in this folder.This email is built from a template consisting of a mix of plain text and substitution variables marked out by caret characters, for example, ^issueId^ and ^title^. Variables are replaced with plain text values before the email is sent.For example, the variables in the following template sentence...
^itemName^ #^issueId^, "^title^," has been ^action^
...are replaced with plain text values, to become...
Issue #1234, "Typo in the New User Interface", has been resolved
You can also add format strings to further modify the substitution variables.Complete documentation on the substitution variables and formats is shown on the template editor page. More details can be found in this topic: Email Template Customization.
Issue Tracking Web Parts
Web parts are the user interface panels that expose selected functionality. If desired, you can create multiple issue trackers in the same folder. Each will track a different set of issues and can have different fields and properties. To do so, create additional issue definitions. When you create each Issues List web part, you will select which issue list to display from a drop down menu.
Issues List Web Part
Once you have created an issue definition in the folder, you can add an interface for your users to a tab or folder.
If you don't provide a web part Name, a generic one based on the name of the issue definition will be used.
Click Submit.
Issues Summary Web Part
The Issues Summary web part displays a summary of the Open and Resolved issues by user. This can be a useful dashboard element for managing balancing of team workloads. Add this web part to a page where needed. You can give it a custom name or one will be generated using the name of the issue definition.Clicking any user Name link opens the grid filtered to show only the issues assigned to that user. Click View Open Issues to see all issues open in the issue tracker.
Note that a given project or folder has only one associated Issues module, so if you add more than one Issues List or Summary web part associated with a given issue definition, both will display the same data.
Related Topics
Tutorial: Issue Tracking: This topic will walk you through the steps to create a new issue definition and issue tracker.
Issues API - Insert and update issues using the API.
Electronic Data Capture (EDC)
LabKey Server lets you design your own surveys and electronic data capture (EDC) tools to support a clinical trial management system. EDC tools can be used to replace the traditional paper process for collecting information from patients or study participants. They also provide higher data quality, by constraining the possible responses, and by removing the error-prone transfer from paper formats to electronic formats.LabKey Server can also pull in data from existing EDC tools and projects, such data collected using REDCap.
Electronic data capture makes collecting information from patients or researchers simple and reproducible. You can craft user-friendly questions that will gather the information you need, mapped to the correct underlying storage system.In order to demonstrate some of the features of the Survey Designer, imagine that you are hosting a conference and wish to gather some advance information about the participants. To collect this information, you will set up a survey that the participants can complete online.
First enable survey functionality in a folder. Each user taking the survey will need to have at least Editor permissions, so you will typically create a new folder such as My Project > Conference Attendee Surveys to avoid wider access to other parts of your project.
Log in to your server and navigate to your "Tutorials" project. Create it if necessary.
If you don't already have a server to work on where you can create projects, start here.
If you don't know how to create projects and folders, review this topic.
Create a new subfolder named "Survey Demo" and accept all defaults.
From the Key Field Name dropdown, select Auto integer key. This will add a primary key field of type "Integer" that will increment automatically to ensure unique values.
Rename the new key field "ParticipantID".
This will make it easier for data from this survey to be integrated with other data about the participants.
Add the additional fields shown in the following image. For each line, click Add Field then enter the Name and Data Type as shown. Be sure not to include spaces in field names.
Click Save.
Survey Design
The next step is to design the survey so that it will request the information needed in the correct types for your results table.The survey designer generates simple input fields labeled with the column names as the default 'questions' presented to the person taking the survey. Note that when you press the button "Generate Survey Questions", the server executes a selectRows API call and pulls in the fields for the default view of the specific table the survey is based on.If a dataset is selected, the field SequenceNum must be either included in the default grid view so the survey question can be generated against it, or the survey JSON code must be manually updated to include sequenceNum. Otherwise, the survey will not be able to save the records submitted.
Return to your folder by clicking the Survey Demo link near the top of the page.
Provide a Label (such as ConferenceAdvance in this example) and Description.
From the Schema dropdown, select lists.
From the Query dropdown, select your list, in this example: SurveyResults.
Click Generate Survey Questions above the entry panel.
Default survey questions are generated using the field information you used when you created your SurveyResults list. Notice the lower left panel contains documentation of the survey configuration options. It is possible to customize the survey at this point, but for now just use the defaults.
Click Save Survey.
You can now use this design to create a survey which users can fill in to populate your results table.
Create and Populate The Survey
Create the Survey:
Saving the survey has returned you to the main folder page.
Add a Surveys web part.
In the Survey Design dropdown: select your survey design (ConferenceAdvance).
Click Submit.
Populate the survey:
Click Create Survey to launch the survey wizard and present the default questions based on your results table.
Enter values for the given fields.
Click Submit Completed Form to finish. There is also a Save button allowing the user to pause without submitting incomplete data. In addition, the survey is auto-saved every 60 seconds.
In practice, each conference participant would go to this folder and click Create Survey to add their own information. As the host, you would view the ConferenceAdvance table and use that data to print name tags, plan seating, etc.You may have noticed that the 'questions' generated based on field names add spaces following CamelCase hints, but are not very helpful and would require external explanation, particularly in the case of the date field we included. In the next step we will explore options for customizing our survey to make it clearer what we are asking.
By default, the survey designer generates a basic entry wizard based on the results list it is designed to populate. Questions are generated for each data type and given the assigned field name field names. Options for building a more user-friendly survey that is more likely to obtain the information you want can be done in several ways:
For a more traditional and user-friendly survey, you can add more description, write survey questions in a traditional sense, and control which fields are optional.
Go back to your survey folder.
Click Edit next to the ConferenceAdvance survey design you created in the previous step. Hover over the row to reveal the (pencil) edit icon.
The main panel contains the JSON code to generate the basic survey. Below the label and description on the right, there is documentation of the configuration options available within the JSON.
Make changes as follows:
Change the caption for the First Name field to read "First name to use for nametag".
Change required field for First Name to true.
Change the caption for the Reception field to read "Check this box if you plan to attend the reception:".
Change the caption for the GuestPasses field to read "How many guests will you be bringing (including yourself)?"
Change the width for the GuestPasses field to 400.
In the Date field, change the caption to "Preferred Talk Date".
Change the hidden paramenter for Title to true (perhaps you decided you no longer plan to print titles on nametags).
Click Save Survey.
Now click Create Survey in the Surveys: ConferenceAdvance web part. You will see your adjusted questions, which should now better request the information you need. Notice that the field you marked required is shown in bold and with an asterisk. Required fields are also listed in a note below the submit button which will be inactive until all required information is provided.
Customize Survey Parameters
You can make various changes to the overall design and layout using parameters to survey outlined in the Survey Configuration Options panel. For example, you can use a card format and change the Survey Label the user sees where they are expected to enter the primary key for your results table, in this case a registration number you gave them to use as participant ID.
Return to your folder and again open the ConferenceAdvance survey design for editing.
In the JSON panel, change "layout" to "card".
Add a new parameter: "labelCaption" : "Attendee Registration Number". All parameters must be comma separated.
Click Save Survey.
In the Surveys: ConferenceAdvance web part, click Create Survey and note the new format and start tab. The bulk of the survey (that will populate the SurveyResults list) is on the second step of the new card-style wizard.
For a complete outline of the survey parameters and their defaults, see:
In addition to the survey question types directly based on field datatypes, there are additional options for more complex questions. For example, for a given text field such as title, you might constrain allowable input by using a lookup from a table of three character abbreviations (Ms., Mr., Mrs, Dr., Hon) to format evenly on nametags. To do so, create a list of allowable options, and add a "Combobox" question to your survey.
First create the list of allowable values. For this example, create list called Titles.
Add a single field called Title of type Text.
Set the Key Field Name to Title.
Save the list, and import the following data:
Title
Mr.
Mrs.
Ms.
Dr.
Click Edit for your ConferenceAdvance survey design.
Open the Question Metadata Examples panel on the right by clicking the small chevron button in the upper right.
Click the name of any question type and an example JSON implementation will appear in a scrollable panel below the bar.
You can cut and paste the JSON from the middle panel in place of one of your default questions. Watch for result type mismatches.
Customize the parameter values as with default questions. In this title lookup example, specify the appropriate containerPath, queryName, and displayColumn.
Save the survey and when you run it again, the field will be presented as a dropdown menu listing all the values from the selected list.
Note that you can have the lookup show only results from a specific custom grid view, such as a filtered subset of the full list, by adding the viewName parameter to the lookup (after queryName).
For a complete list of question datatypes and their parameters, see:
The following properties are available when customizing a survey to adjust layout, style, question formatting, and defaults. This documentation is also available directly from the survey designer on the Survey Configuration Options panel.
The indenting of the 'Parameter' column below shows the structure of the JSON objects.
Parameter
Datatype
Description
Default Value
beforeLoad
object
Object to hold a javascript function.
fn
string
A javascript function to run prior to creating the survey panel. Useful for loading custom scripts, the specified function is called with two parameters : callback and scope which should be invoked after the furnished function has run, for example: "fn": "function(callback, scope){ LABKEY.requiresScript('myscript.js', callback, scope); }"
footerWiki
object
Configuation object for a wiki that will be displayed below the survey panel.
containerPath
string
Container path for the footer wiki. Defaults to current container.
current container path
name
string
Name of the footer wiki.
headerWiki
object
Configuration object for a wiki that will be displayed above the survey panel.
containerPath
string
The container path for the header wiki. Defaults to current container.
current container path
name
string
Name of the header wiki.
layout
string
Possible values: • auto - vertical layout of sections • card - wizard layout
auto
mainPanelWidth
integer
In card layout, the width of the main section panel.
800
navigateOnSave
boolean
Set true to navigate away from the survey form after the save action. Navigation will take the user to the returnUrl, if provided, or to the project begin action..
false
autoSaveInterval
integer
The interval in milliseconds that auto-saving of the survey results will occur.
60000
disableAutoSave
boolean
Set to true to disable autosaving of survey results. This setting will take precedence over the autoSaveInterval value.
false
sections
array
An array of survey section panel config objects.
border
boolean
Set to true to display a 1px border around the section. Defaults to false.
false
collapsed
boolean
If layout is auto, set to true to begin the section panel in a collapsed state.
false
collapsible
boolean
If layout is auto, set to true to allow the section panel to be collapsed. Defaults to true.
true
defaultLabelWidth
integer
Default label width for questions in this section.
350
description
string
Description to show at the beginning of the section panel.
extAlias
string
For custom survey development, the ext alias for a custom component.
header
boolean
Whether to show the Ext panel header for this section.
true
initDisabled
boolean
In card layout, disables the section title in the side bar. Defaults to false.
false
layoutHorizontal
boolean
If true, use a table layout with numColumns providing the number of columns.
false
numColumns
integer
The number of columns to use in table layout for layoutHorizontal=true.
1
padding
integer
The padding to use between questions in this section. Defaults to 10.
10
questions
array
The array of questions. Note that the 'questions' object is highly customizable because it can hold ExtJS config objects to render individual questions.
extConfig
object
An ExtJS config object.
required
boolean
Whether an answer is required before results can be submitted.
false
shortCaption
string
The text to display on the survey end panel for missing required questions.
hidden
boolean
The default display state of this question (used with listeners).
false
listeners
object
JavaScript listener functions to be added to questions for skip logic or additional validation (currently on 'change' is supported).
auto
change
object
Listener action.
question
string or array
Name(s) of parent question(s). Must be all lower case.
fn
string
JavaScript function to be executed on parent.
title
string
The title text to display for the section (auto layout displays title in header, card layout displays title in side bar).
showCounts
boolean
Whether to show count of completed questions.
false
sideBarWidth
integer
In card layout, the width of the side bar (i.e. section title) panel.
250
start
object
Configuration options for the first section of the survey.
description
string
Description appears below the 'survey label' field, in the start section, i.e., the first section of the survey.
labelCaption
string
Caption that appears to the left of the 'survey label' field, defaults to "Survey Label".
"Survey Label"
labelWidth
integer
Pixels allotted for the labelCaption.
sectionTitle
string
The displayed title for the start section, defaults to "Start".
"Start"
useDefaultLabel
boolean
If true, the label field will be hidden and populated with the current date/time.
false
Survey Designer: Examples
This topic contains some example survey questions to suit specific scenarios. You can also find the JSON metadata structure built in for each type of question below.
The following example assumes you have a List named "Participants" with the following fields:
SSN (this is your Primary Key, which is a string)
FirstName
LastName
Phone
When the List contains records, and a user enters a matching SSN number, the remaining fields in the survey will auto populate with data from the matching record.Download the JSON code for this survey: survey-SSNautopop.json
Auto Populate with the Current User's Email
When a user changes the Patient Id field, the email field is auto populated using the currently logged on user's info.Download the JSON code for this survey: survey-Emailautopop.json
Hidden Radio Group
When Option A is selected, a hidden radio group is shown below.Download the JSON code for this survey: survey-hiddenRadioGroup.json
Hidden Question
A hidden question appears when the user enters particular values in two previous questions. In this example, when the user enters 'Yes' to questions 1 and 2, a 3rd previously hidden question appears.Download the JSON code for this survey: TwoQuestions.json
Radio Buttons / Rendering Images
The following example renders radio buttons and an image. The 'questions' object holds an 'extConfig' object, which does most of the interesting work.The 'start' object hides the field "Survey Label" from the user.Download the JSON code for this survey: survey-RadioImage.json
Likert Scale
The following example offers Likert scale questions as radio buttons.Download the JSON code for this survey: survey-Likert.json
Concatenate Values
When two fields are filled in, they auto-populate another field as a concatenated value.Download the JSON code for this survey: survey-Concat.json
Checkbox with Conditional/Skip Logic
Two boxes pop-up when the user picks Other or Patient Refused. The boxes are text fields where an explanation can be provided.Download the JSON code for this example: survey-Skip1.json
Text Field (w/ Skip Logic)
Shows conditional logic in a survey. Different sets of additional questions appear later in the survey, depending on whether the user enters "Yes" or "No" to an earlier question.Download the JSON code for this example: survey-Skip2.jsonThe JSON metadata example for the 'DaysSick' field, for example, looks like this:
The following example includes a question answered using a dropdown to select a time in 15 minutes increments.Download the JSON code for this example: survey-timedropdown.json
Calculated Fields
The following example calculates values from some fields based on user entries in other fields:
Activity Score field = the sum of the Organ/Site scores, multiplied by 2 if Damage is Yes, plus the Serum IgG4 Concentration.
Total number of Urgent Organs = the sum of Urgent fields set to Yes.
Total number of Damaged Organs = the sum of Damaged fields set to Yes.
The Label field loads data into the Label field of the surveys.Surveys table. To use an autogenerated value for the Label, see below. Autogenerated values are a modified timestamp, for example, "2017-08-16T19:05:28.014Z".Add the following start object before the sections to hide the "Label" question.
A Combobox (Lookup) question will give the user a dropdown selector populated from a list in a given container. If you want to have the dropdown only contain a filtered subset of that list, such as only 'in use' values or some other constraint, first create a custom grid view of that list that contains the subset you want. Then add the viewName parameter to the combobox "lookup" definition.For example, if you have a list named "allSOPs" with a 'retired' field (set to true when an SOP is outdated) you could create a grid view that filtered to show only those which have not been marked 'retired' and name it, for example "activeSOPs". A combobox lookup showing that list might look like this:
The user would see only the list items in that specific custom grid view.
Include Tooltips or Hints for Questions
To have a survey question show a tooltip or hover hint, you can make use of an extConfig object following this example. Include a labelAttrTpl attribute set to provide the tooltip text as the data-qtip:
"questions": [ { "extConfig": { "width": 800, "xtype": "textfield", "name": "test", "labelAttrTpl": " data-qtip='This tooltip should appear on the field label'", "fieldLabel": "Field with tooltips" } } ],
By default, the survey designer will autosave submissions in progress every minute (60000 milliseconds). If desired, you can disable this auto-save behavior entirely, or enable it but change the interval to something else. The relevant parameters are included in the JSON for the survey itself.
autosaveInterval: The interval (in milliseconds) that auto-saving of the survey results will occur. The default value is : 60000.
disableAutoSave: Set true to disable autosaving entirely. This setting takes precedence over the autoSaveInterval value and defaults to false.
A few examples follow.To disable autosave, you would start the definition with:
Premium Feature — Available in the Professional and Enterprise Editions of LabKey Server. Also available as an Add-on to the Starter Edition. Learn more or contact LabKey.
LabKey Server can import data collected using REDCap (Research Electronic Data Capture) online surveys and forms. Existing REDCap projects can be directly imported into a LabKey Server study using the REDCap API. You can either import the REDCap data into an existing study, or use it as the demographic basis for a new study.
REDCap data objects are imported into LabKey Server as follows:
REDCap data object
LabKey Server data object
Notes
form00000000000000
dataset0000000000000000000000
Forms are imported as LabKey Server study datasets. You can specify which forms should be imported as demographic datasets using the configuration property demographic.
event
visit or date
Events are imported as either LabKey Server visits or dates. You can specify 'visit' or 'date' import using the configuration property timepointType.
multi-choice fields
lookups
Multiple choice fields are imported as lookups. Possible values are imported into Lists. The name of the created List either follows the naming convention <Multi-Choice-Field-Name>.choices OR, if multiple tables use the same multiple choices, a common list named like "shared_1.choices" will be created. Note that multi-choice fields names longer than 56 characters cannot currently be imported, because LabKey limits the length of List names to 64 characters.
You can also set up data reloading on a recurring schedule to capture new data in the REDCap project.
Notes:
REDCap forms must have the status complete to be imported. If forms are marked 'incomplete,' the data structure will be created, but it will be empty.
If you are importing into a visit-based study, REDCap must either include events or set a default event. Date-based studies require choosing a date field for each dataset.
Shared Lookup Lists
When a REDCap import including multi-choice files would create duplicate lists in LabKey, such as when many questions have a Yes/No answer, the system will only create a single lookup list for all of the lookups, containing the values used.LabKey will consider the list the same if:
Key/values match exactly.
Key/values can appear in any order, i.e. the matching determination is not order dependent.
All key/value pairs must match. We do not consider subsets as a match.
In your study folder, go to: > Folder > Management and click the Folder Type tab.
Under Modules, place a checkmark next to REDCap.
Click Update Folder.
Connect and Configure External Reloading
In your study folder, click the Manage tab.
Click Manage External Reloading.
Click Configure REDCap.
Configure connections and reloading on the three tabs:
Connection - Use this tab to specify the REDCap server you intend to draw data from. You can also define a recurring schedule to automatically reload the data.
Datasets - Use this tab to specify import behavior for individual REDCap forms. For example, you can import a given form into LabKey as either a dataset (the default behavior) or as a List. You can specify the timepoint type (visit or date), and additional keys when required.
Advanced - Use this tab to define advanced configurations using XML.
Connection Tab
The information you enter here sets up communication between LabKey Server and a REDCap server (or servers). For each REDCap project you wish to load data from, there must be a separate row of connection information.
XML Export Format: If your REDCap project supports the CDISC format, confirm that this is checked to proceed. Remove this checkmark if your REDCap project does not support the CDISC format or if you are using a REDCap version before 6.12. Documentation for the earlier non-CDISC format is available in the documentation archive at REDCap Legacy Import.
REDCap Server URL: The URL of the target REDCap server. The URL should end with 'api/', for example: https://redcap.myServer.org/api/
Change Token: Check the box to change the authorization token.
Token: A hexidecimal value used by the REDCap server to authenticate the identity of LabKey Server as a client. Get the token value from your REDCap server, located on the REDCap API settings page of the project you are exporting from.
Enable Reloading: (Optional). Place a checkmark here to start reloading on a schedule defined below.
Load On: Optional. Set the start date for the reloading schedule.
Repeat (days): Optional. Repeat the reload after this number of days.
Action Buttons
Save: Create a connection to the REDCap server. After clicking Save you can refine the import behavior on the Datasets tab. For details see Medidata / CDISC ODM Integration.
Reload Now: Loads the data from REDCap on demand.
Datasets Tab
Click the Datasets tab to configure study properties and per-dataset import behavior.
Click the (pencil) icon.
Here you can edit settings including:
Whether the target study is visit or date based.
For date based studies, you can specify the name of the column containing the date.
For visit based studies, you can provide a default sequence number.
Whether to import all datasets with shared defaults or add per-dataset definitions for:
Whether the dataset is demographic, clinical, or has a third key.
Whether it should be imported as a list (instead of a dataset).
Which column to use as the date field or what the default sequence number should be (if these differ from the study-wide settings on the previous page.
The Advanced tab lets you control advanced options using XML. After completing the Datasets tab, XML will be auto-generated for you to use as a template.For details about the namespace and options available, review the REDCap XSD schema file.
Note that the XML used to configure REDCap and that used to configure Medidata/CDISC ODM reloads are not inter-compatible.
Available configuration options are described below:
projectName: Required. The name of the REDCap project (used to look up the project token from the netrc file). The projectName must match the project name entered on the Authentication tab.
subjectId: Required. The field name in the REDCap project that corresponds to LabKey Server's participant id column.
timepointType: Optional. The timepoint type (possible values are either 'visit' or 'date'), the default value is: 'date'.
matchSubjectIdByLabel: Optional. Boolean value. If set to true, the import process will interpret 'subjectId' as a regular expression. Useful in situations where there are slight variations in subject id field names across forms in the REDCap project.
duplicateNamePolicy: Optional. How to handle duplicate forms when exporting from multiple REDCap projects. If the value is set to 'fail' (the default), then the import will fail if duplicate form names are found in the projects. If the value is set to 'merge', then the records from the duplicate forms will be merged into the same dataset in LabKey Server (provided the two forms have an identical set of column names).
formName: Optional. The name of a REDCap form to import into LabKey Server as a dataset.
dateField: Optional. The field that holds the date information in the REDCap form.
demographic: Optional. Boolean value indicating whether the REDCap form will be imported into LabKey Server as a 'demographic' dataset.
In some situations where your REDCap data has many columns, and where there are many lookups, you may run into the limits of the number of columns on the underlying database. In such a case you might see an error like:
org.postgresql.util.PSQLException: ERROR: target lists can have at most 1664 entries
In order to avoid this error you can potentially restructure the data in order to reduce the resulting width. If that is not possible, you can choose to suppress the creation of lookups entirely for the study. Note that this is a significant change and should only be used when other alternatives cannot be used.To suppress linking, set disableLookups to true in the XML for the cdiscConfig on the Advanced tab. For example:
Premium Feature — Available in the Professional and Enterprise Editions of LabKey Server. Learn more or contact LabKey.
LabKey Server can connect directly with, and import data from, Medidata Rave servers. The data is exported directly from the server via the Medidata API; direct CDISC file import is not supported.
LabKey Server uses Medidata Rave's webservice interface to import both data and metadata. The data is imported into study datasets, where it can be integrated with other data in the study environment. Data synchronization can be performed manually on demand or can be configured to occur automatically on a repeating schedule. The overall behavior is similar to that provided for REDCap, where forms are mapped to study datasets on LabKey Server.In detail, import of Medidata Rave data goes as follows: data and metadata are exported from the Medidata Rave server using their webservice interface. LabKey Server transforms the data into LabKey's archive format which is then passed to the data processing pipeline for import into a study container. If the target datasets (or lists) do not already exist, they are created, then new records are inserted, and if the targets existed previously, any existing records are updated.
Configure Data Import
Importing from a Medidata Rave server assumes that you already have a LabKey study in place, with the CDISC_ODM module enabled.To set up data import from a Medidata server, follow the instructions below:
In your study folder, click the Manage tab and click Manage External Reloading.
Click CONFIGURE CDISC ODM.
Connection Tab
Fill out the initial form:
Study Metadata URL: The URL for the metadata web service.
Study Data URL: The URL for the data web service.
Change Password: Check the box to enable entry of username and password.
User Name: A user who has access to the Medidata server.
Password: The password for the above user.
Enable Reloading: Select for automatic reloading on a repeating schedule.
Load on: The start date if automatic reloading is selected.
Repeat (days): LabKey will iterate the data reloading after these many days.
On the Edit Form Configuration panel, enter the following information:
Study Label: Enter your LabKey Server target study label. (This is set at Manage tab > Study Properties.)
timepoint type: Select VISIT or DATE. This much match the timepoint style for the target study.
date field: If DATE is selected above, select the default name for date fields in your datasets. This can be overridden for individual non-conforming datasets.
default sequence number: If VISIT is selected above, you can opt to specify a default sequence number to be used for any data that does not have a corresponding event defined. This value can also be overridden on a per-form basis.
additional key field: Select the default additional key field, if any. This can be overridden for individual non-conforming datasets.
Import all datasets: If selected, all CDISC forms will be imported by default, although the next panel lets you exclude individual datasets. If unselected, the next panel lets you select which individual datasets to import.
Click Next
Note that advanced users can skip this next panel and instead choose to edit the configuration XML directly, if this is done this panel will be updated to reflect the state of the XML.
Use this panel to select which CDISC forms to import, and how they are imported.
Selecting a form for import reveals additional options for how to import it to LabKey Server.
Selecting demographic causes the form to be imported to a LabKey demographic dataset.
Unselecting demographic causes the form to be imported to a LabKey clinical dataset.
Selecting export as list causes the form to be imported to a LabKey list. Choose this option for forms that do not conform to dataset uniqueness constraints.
date field: When the timepoint type is DATE, you will see this field and can use the dropdown to override the default date field indicated in the previous panel.
default sequence number: When the timepoint type is VISIT, you will see this field and can use the dropdown to override the default sequence number indicated in the previous panel.
additional key field: Use this dropdown to override the default third key field indicated in the previous panel.
Click Apply. (This will update the XML on the Advanced tab.)
On the CDISC ODM Configuration page, click Save.
To manually load the data, click the Connection tab and then Reload Now.
Advanced Tab
The Advanced tab lets you control advanced options using XML. We recommend that you allow the UI to auto-generate the initial XML for you, and then modify the generated XML as desired. You can auto-generate XML by working with the Datasets tab, which will save your configuration as XML format on the Advanced tab.Note that the REDCap feature and the CDISC feature do not use compatible XML schemas and you cannot swap configuration between them.For details, see the CDISC XSD schema file.Example XML:
Error: "Element redcapConfig@http://labkey.org/study/xml/redcapExport is not a valid cdiscConfig@http://labkey.org/cdisc/xml document or a valid substitution"This error occurs when you configure a CDISC ODM reload using REDCap-specific XML elements.Solution: Use the UI (specifically the Datasets tab) to autogenerate the XML, then modify the resulting XML as desired.
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
This topic describes how to load clinical data in the CDISC ODM XML file format into a LabKey Study. For example, data collected using DFdiscover can be exported in this format. The data loading process is based on a File Watcher processing pipeline configured to read CDISC_ODM XML files.
Note that while this topic is using the DFdiscover example, other EDC tools that can output data in a CDISC XML format can also be integrated using this method.
Process Overview
A summary of the data loading process:
Data is collected and stored using a tool like DFdiscover.
The data is manually exported into XML files that conform to the CDISC ODM format.
These XML files are dropped into a location watched by LabKey Server.
LabKey Server detects the files, which triggers the data processing pipeline.
The pipeline parses the XML files and loads the data into study datasets or lists, as specified.
File Watcher Options
This File Watcher supports two modes of watching for content in a specified location.
Trigger on individual file(s) (typically conforming to a naming pattern provided) and then act on each file.
Trigger on addition of a new folder, acting on the contents of that folder in a batch. Under this mode, it is important that the files required are already in the folder when it is dropped into the watched location. In a case where a user manually creates a subfolder, then drops files into it there, it is likely that the trigger will fire before all the content is present, leading to unexpected results.
In both cases, the "quiet period" must be long enough for completion of the entire upload to be acted upon.The Import Study Data from a CDISC ODM XML File integration can use either the single file method (in which case a default configuration is applied) or the use of the folder approach in order to specify the configuration in a separate file uploaded with the data.
Configuration File Options
Default Configuration
In the mode where only the data files are provided to the watched location, the
default configuration is used:
All forms in the data file are imported as datasets
Customized Configuration
When the data file(s) and a configuration file are provided together in a folder, that configuration file can contain different options, such as importing some files as lists or with specific key field settings etc. You will specify the name of this configuration file in the File Watcher definition, and it must conform to the CDISC XML Schema.
If you configure the File Watcher to look for a configuration file, it will need to be placed in the same folder as the target file. If the configuration file is not found, the File Watcher will error out.
Best practice is to upload the configuration file before the target file OR set the quiet period long enough to allow enough time to upload the configuration file.
If the File Watcher is configured to move files to another container, only the target file will be moved automatically and the job will fail. In this case you will want to upload a folder with both the configuration and target XML files in the folder. To set up, either don't provide a file name expression at all or make sure the expression will accept the folder name(s).
This File Watcher trigger has a special file filter that accepts a target folder as long as it contains one or more XML files.
An example custom configuration file, giving different configurations for a set of forms: Click here to download.
To set up the File Watcher pipeline, follow these steps:
Create or identify a study folder where you wish to load the DFdiscover data.
Navigate to this study folder.
Select > Go To Module > FileContent.
In the file repository, create a directory where the server will watch for files/folders. In this example, we named the directory "watched".
Go to > Folder > Management and click the Folder Type tab.
Check the box for CDISC_ODM and click Update Folder to enable the module.
Next, determine whether you need to customize the configuration in a file and if necessary, create that file. Both options are shown in these File Watcher configuration steps:
Return to > Folder > Management and click the Import tab.
Configure a File Watcher that uses the task Import Study Data from a CDISC ODM XML File.
In this example, we have a file browser directory named "watched" to watch and when we upload our files, want to move the files to "@pipeline/processed_files" for processing.
Note that the Quiet Period should be set for enough seconds to allow the data and configuration files to be uploaded.
For a data file only configuration, provide a file pattern and do not name a configuration file:
For a data and config file configuration, do not provide a file pattern (you will upload both files in a folder together) and do provide the name of your config file.
Use the File Watcher
If you are using the data file only method, drop the .xml file into the watched location.
If you are importing both a data and config file, on your filesystem, create a folder containing both files.
Drop this folder into the "watched" location. You'll want to be sure that the quiet period is sufficient for the entire contents to upload.
For either method, the File Watcher will trigger and you'll see the data imported into your study as directed.
LabKey Server's adjudication module is a legacy feature that requires significant customization and assistance. It is not included in standard LabKey distributions.Please contact LabKey to inquire about support options.
Adjudication is a workflow process in which two (or more) independent people (or teams) make a determination about diagnoses given certain data and criteria. Each team of adjudicators has access to the same data, but can not see the determinations made by others adjudicators until all determinations are complete.
Documentation
Learn about the adjudication module in the documentation archives:
This topic covers two ways to manage contact information for project users. Individuals may manage their own contact information, and admins can add a Project Contacts web part to show the information for all users of that project.
The Project Contacts web part displays contact information for all users with active accounts who are members of one or more project-level security groups for the current project. Note that this is not a specific project group, but the sum of users in all project groups.Add this web part in page admin mode. Select Contacts from the web part selector and click Add. Learn about adding web parts in this topic: Add Web Parts.Administrators have access to view this webpart, and can grant access to users who are not admins (or groups of such users) by assigning the See User and Group Details site permission.
Adding Project Users
All project users can be viewed in the web part and via > Folder > Project Users, but new project users cannot be added using either of these grids.To add new project users, access > Folder > Permissions, create a new project-level security group and add users to it.
Edit Contact Information
Each user can enter their own information in their account details.To access your contact information:
Make sure you are logged in to the LabKey Server installation.
Open the pulldown menu showing your username in the top right corner of the page.
Select My Account to show your contact information.
Click Edit to make changes.
You can edit your contact information from this page, except for your email address. Because your email address is your LabKey user name, you can't modify it here. To change your email address, see My Account.
A full list of LabKey publications is available here.
Development
Developer Resources
LabKey Server is broadly API-enabled, giving developers rich tools for building custom applications on the LabKey Server platform. Client libraries make it easy to read/write data to the server using familiar languages such as Java, JavaScript, SAS, Python, Perl, or R.Stack diagram for the LabKey Server Platform:
To use most of the features in these sections, you must have the "Platform Developer" or "Trusted Analyst" roles on your server. Learn more in Developer Roles.
Client API Applications
Create applications by adding API-enhanced content (such as JavaScript) to wiki or HTML pages in the file system. Application features can include custom reports, SQL query views, HTML views, R views, charts, folder types, assay definitions, and more.
LabKey Client APIs - Write simple customization scripts or sophisticated integrated applications for LabKey Server.
Tutorial: Create Applications with the JavaScript API - Create an application to manage reagent requests, including web-based request form, confirmation page, and summary report for managers. Reads and write to the database.
JavaScript API - Select data from the database and render as a chart using the JavaScript API.
Scripting and Reporting
LabKey Server also includes 'hooks' for using scripts to validate and manipulate data during import, and allows developers to build reports that show data within the web user interface
Transform Scripts for Assays (which operate on the full file) can be written in virtually any language.
Trigger Scripts which operate at the row-level, are written in JavaScript and supported for most data types.
Using R scripts, you can produce visualizations and other analysis on the fly, showing the result to the user in the web interface.
LabKey Open Source Project - An overview of the improvement process for LabKey Server and how you can contribute.
Set Up a Development Machine
This topic provides step-by-step instructions to assist developers in acquiring the LabKey Server source code, installing required components, and building LabKey Server from source. The instructions here are written for a Windows machine. To set up development on a Mac/OSX machine, use these instructions in conjunction with the topic: Set up OSX for LabKey Development.
The LabKey Server source files are stored in multiple GitHub repositories. In these steps, you'll establish and name a new local enlistment location on your machine, which we refer to here as <LK_ENLISTMENT>. One option is to use "C:\labkey\labkeyEnlistment".
Problem executing request for repository data with query 'query { search(query: "org:LabKey fork:true archived:false ", type:REPOSITORY, first:100
Clone Repositories
Start by cloning the server repository from GitHub to create the <LK_ENLISTMENT> location. Then navigate to it and add the core repositories (and any additional modules required).
Begin a level "above" where you want the <LK_ENLISTMENT>. For example, you might start in a "C:\labkey\" directory.
Clone the server repository from git into a directory with your <LK_ENLISTMENT> name, such as labkeyEnlistment.
Note: LabKey uses Artifactory to maintain the premium modules of LabKey Server. If you are using a Premium Edition, please follow these setup instructions for your Artifactory account.
Development Machine Directory Structure
The basic hierarchy of a development machine should look like this. Many files and directories are not shown here.
By default, the develop branch is checked out when a new repository is cloned. If you would like a different branch you will need to check out the desired branch in each repository as well as the root of your enlistment. You must keep all parts of your enlistment in sync. For example, to change a the default/minimal set of repositories to the release24.11 branch, start from LK_ENLISTMENT:
LK_ENLISTMENT> cd server/modules/platform LK_ENLISTMENT\server\modules\platform> git checkout release24.11 LK_ENLISTMENT\server\modules\platform> cd ../commonAssays LK_ENLISTMENT\server\modules\commonAssays> git checkout release24.1 LK_ENLISTMENT\server\modules\commonAssays> cd ../../.. LK_ENLISTMENT> git checkout release24.11
You can simultaneously check out all of your repositories to a given branch by using both of the following commands from the root of your enlistment in this order. However, you cannot use these commands until after you have set the environment variable GIT_ACCESS_TOKEN:
The LabKey build depends on Node.js and the node package manager (npm). The build process automatically downloads and installs the versions of npm and node that it requires. You should not install npm and node yourself. For details on the Node.js dependency and build-prod script requirement, see Node.js Build Dependency.
Gradle Configuration
Please note that you do not need to install Gradle. LabKey uses the gradle wrapper to make updating version numbers easier. The gradle wrapper script (either gradlew or gradlew.bat) is included and is already in the <LK_ENLISTMENT> directory.However, you must:
Create a globalgradle.properties file that contains global settings for your Gradle build. Your enlistment includes a template file to use.
Also edit the separate <LK_ENLISTMENT>/gradle.properties file to enable options required for developing locally.
Global gradle.properties file for your user account:
Create a .gradle directory in your home directory.
On Windows the home directory is typically: C:\Users\<you>.
Create a .gradle folder there.
If the file explorer does not allow you to create a folder beginning with a period, you may need to navigate to that location and use a command prompt to type:
mkdir .gradle
Create a gradle.properties file in that new location using the following process:
Copy the file <LK_ENLISTMENT>/gradle/global_gradle.properties_template to C:\Users\<you>\.gradle.
Rename the file so it is named just gradle.properties.
Create or modify the system environment variable JAVA_HOME so it points to your JDK installation location (for example, C:\labkey\apps\jdk-##).
PATH
Add the JDK to your system PATH. Using the JAVA_HOME variable will simplify upgrades later:
%JAVA_HOME%\bin
Add the <LK_ENLISTMENT>\build\deploy\bin location to your system PATH. This directory does not exist yet, but add it to the path anyway. The Gradle build uses this directory to store binaries used by the server, such as graphvis, dot, and proteomics tools:
<LK_ENLISTMENT>\build\deploy\bin
For example:
C:\labkey\labkeyEnlistment\build\deploy\bin
Set Up the LabKey Project in IntelliJ
The LabKey development team develops LabKey using IntelliJ IDEA.
Install IntelliJ
Download and install the latest version of IntelliJ IDEA. Either the Community or Ultimate Edition will work.
Note that if you are developing against an older version of LabKey, you may need to use a version of IntelliJ that was available at the time of that release. We do not test newer releases of IntelliJ with older versions of LabKey.
Configure the LabKey Project in IntelliJ
Run the following gradle commands (in the <LK_ENLISTMENT> directory) to configure the workspace and other default settings:
gradlew ijWorkspaceSetup gradlew ijConfigure
Open the LabKey project in IntelliJ:
Launch IntelliJ.
If your IntelliJ install is brand new, you will see the "Welcome to IntelliJ" pop up screen. Click Open.
If you have previously installed IntelliJ, select File > Open.
Select your <LK_ENLISTMENT> directory. You may need to confirm you trust the directory.
If you are developing with TypeScript, add a Path Variable. Most users will not need to do this.
Click the four-bars menu in the upper left, the select File > Settings > Appearance & Behavior > Path Variables.
Click the plus icon in the upper right.
Add a NODE_PATH variable and set it to the path to the node binary that is installed by the build, for example, <LK_ENLISTMENT>\build\.node\node-v#.#.##\bin\node
Click OK to close the Settings window.
Configure the Target JDK
In IntelliJ, select File > Project Structure.
Under Project Settings, click Project.
Under SDK select Add JDK...
Browse to the path of your JDK, for example, (C:\labkey\apps\jdk-##), and click OK.
Click Edit. Change the name of the JDK to "labkey".
Click OK to close the Project Structure window.
Open the Gradle tool window at View > Tool Windows > Gradle.
Click the refresh icon. This will take as much as 5-20 minutes. You should start seeing messages about its progress. If not, something is probably hung up. Wait for this sync to complete before progressing with further IntelliJ configuration steps.
Set the Run/Debug Configuration by opening the "Run" menu. It may initially read "GWT" or something else.
Users of IntelliJ Community should select LabKey Embedded Tomcat Dev.
Users of IntelliJ Ultimate should select Spring Boot LabKey Embedded Tomcat Dev.
Be sure that IntelliJ has enough heap memory. The default max is OK if you’re just dealing with the core modules, but you will likely need to raise the limit if you’re adding in custom modules, 3GB seems sufficient.
You may also want to increase the number of open tabs IntelliJ will support. The default is 10, and depending on your working process, you may see tabs disappear unexpectedly. Select Window > Editor Tabs > Configure Editor Tabs... and scroll down to the Closing Policy section to increase the number of tabs you can have open in IntelliJ at one time.
Build and Run LabKey
Configure the pg.properties File
The LabKey source includes a configuration file for use with PostgreSQL (pg.properties), specifying JDBC settings, including URL, port, username, password, etc.
Note that for users of Microsoft SQL Server, the file will be mssql.properties and the task for configuration will be "pickMSSQL" instead of "pickPg".
Open the file <LK_ENLISTMENT>/server/configs/pg.properties
Edit the file, adding your values for the jdbcUser and jdbcPassword. (This password is the one you specified when installing PostgreSQL.)
Run pickPg
In a command window, go to <LK_ENLISTMENT>
Run this task to configure the application.properties file:
gradlew pickPg
When you run 'pickPg', the values that you've provided in both <LK_ENLISTMENT>/configs/application.properties and the pg.properties file in the same location will be copied into the <LK_ENLISTMENT>/build/deploy/embedded/config/application.properties file, overwriting previous values. It is important to remember that you will need to repeat this task any time you change values in either of these properties files.
On the command line, go to the <LK_ENLISTMENT> directory, and invoke the gradle build task:
gradlew deployApp
To control which modules are included in the build, see Customize the Build.
Run LabKey Server
To run and debug LabKey:
Click either the or icon to run or debug either 'Spring Boot LabKey Embedded Tomcat Dev' or 'LabKey Embedded Tomcat Dev' (depending on your edition of intelliJ). Hover text will guide you.
Once LabKey Server starts up successfully, navigate your browser to http://localhost:8080/ to begin debugging.
You should see the LabKey application and be invited to create a new user account the first time you start up using a new database.
While you are debugging, you can usually make changes, rebuild, and redeploy LabKey to the server without stopping and restarting Tomcat. Occasionally you may encounter errors that do require stopping and restarting Tomcat.
Optional Additions
Test Automation
To be able to run automated tests, you also need to clone the testAutomation repository into the server directory under <LK_ENLISTMENT>. Do not clone this repo into the 'modules' directory.:
Many optional modules are available from the LabKey repository on GitHub. To include these modules in your build, install a Git client and clone individual modules into the LabKey Server source.
To add a GitHub module to your build, clone the desired module into <LK_ENLISTMENT>/labkey/server/modules. For example, to add the 'reagent' module:
Note that you can get the URL by going to the module page on GitHub (for example, https://github.com/LabKey/reagent), clicking Clone or Download, and copying the displayed URL.
Manage GitHub Modules via IntelliJ
Once you have cloned a GitHub module, you can have IntelliJ handle any updates:To add the GitHub-based module to IntelliJ (and have IntelliJ generate an .iml file for the module):
In IntelliJ, open the Gradle tool window at View > Tool Windows > Gradle.
Refresh the Gradle window by clicking the arrow circle in the upper left of the Gradle window
To update the GitHub-based module using IntelliJ:
To have IntelliJ handle source updates from GitHub, go to File > Settings (or IntelliJ > Preferences).
Select Version Control.
In the Directory panel, scroll down to the Unregistered roots section, select the module, and click the Plus icon in the lower left.
In the Directory panel, select the target module and set its VCS source as Git, if necessary.
Note that IntelliJ sometimes thinks that subdirectories of the module, like module test suites, have their sources in SVN instead of Git. You can safely delete these SVN sources using the Directory panel.
To sync to a particular GitHub branch: in IntelliJ, go to VCS > Git > Branches. A popup menu will appear listing the available Git modules. Use the popup menu to select the branch to sync to.
If you have added a new module to your enlistment, be sure to customize the build to include it in your Gradle project and then refresh the Gradle window to incorporate it into IntelliJ, as described above.
Generate Reference Docs
To generate documentation, such as the XML reference and Javadoc, clone the tools repository into the enlistment root (to create a sibling of the /server directory):
LabKey includes an embedded version of Tomcat 10. To update a development machine to use embedded Tomcat for the first time, follow these steps:
Get the latest changes from all key repos (server, platform, etc.) with git pull. Don't forget to also update the root of your enlistment. Using the two source upgrade steps below is recommended.
Update IntelliJ with the latest run configurations with gradlew ijConfigure
Within your <LK_ENLISTMENT> level gradle.properties file, uncomment the property useLocalBuild
gradlew cleanBuild
gradlew pickPg (or gradlew pickMSSQL)
gradlew deployApp
Within IntelliJ, do a Gradle Refresh in the Gradle Window.
Update Java
To update the JDK in the future, follow these steps.
Update your system variable JAVA_HOME to point to the new location, unless you changed folder names so that the location path is the same.
Update the IntelliJ > File > Project Structure > SDK > labkey home path to point to the new location. Rename the existing "labkey" to its version number. Add the new JDK, then rename the new version "labkey".
Add additional flags to the VM options section. For example, Java 17 requires these flags in your JVM switches:
Over time, you will want to update IntelliJ to stay current with features and enhancements. You can use the Help > Check for updates... option from within the application. Watch for installation alerts about any .idea files you may have edited. Note that updates may be incremental and after updating, a second more recent update may be available.After updating, recheck the IntelliJ settings described in the above section.Note that if you are developing against an older version of LabKey, you may need to use a version of IntelliJ that was available at the time of that release. We do not test newer releases of IntelliJ with older versions of LabKey.
Update LabKey Source Code
If you're working against the develop branch, you will continuously use "Update Project" to keep your local enlistment in sync. Where this option is in IntelliJ varies by version. It may be represented with a down-left-arrow, on the "four bars > Git" menu or accessed using a shortcut like ctrl-T.If you have been synchronized to a specific branch (such as "release23.11"), you can change to a newer branch by using the recursive gitCheckout option. Use "Update Project" first, then:
Premium Resources AvailableSubscribers to premium editions of LabKey Server can learn more about developing with LabKey with the developer topics and code examples listed on this page:
In the steps below, we use <LK_ENLISTMENT> to refer to the directory into which you checked out your enlistment (i.e., the parent of the server directory).
Your First Gradle Commands
1. Execute a gradle command to show you the set of currently configured projects (modules). You do not need to install gradle and should resist the urge to do so. We use the gradle wrapper to make updating version numbers easier. The gradle wrapper script (either gradlew or gradlew.bat) is included in the enlistment and is already in the <LK_ENLISTMENT> directory.
On the command line, type ./gradlew projects (Mac/Linux) or gradlew projects (Windows)
2. Execute a gradle command to build and deploy the application
./gradlew :server:deployApp
This will take some time as it needs to pull down many resources to initialize the local caches. Subsequent builds will be much faster.
Changing the Set of Projects
Gradle uses the <LK_ENLISTMENT>/settings.gradle file to determine which projects (modules) are included in the build. By default, all modules included in the <LK_ENLISTMENT>/server/modules are included. To exclude any of these modules, or otherwise customize your build, you will need to edit this file. See the file for examples of different ways to include different subsets of the modules.
Source directory - Where the source code lives. This is where developers do their work. The build process uses this as input only.
Build directory - Where all the building happens. This is the top-level output directory for the build. It is created if it does not exist when doing any of the build steps.
Module build directory - Where the building happens for a module. This is the output directory for a module's build steps. It is created if it does not exist when doing a build.
Staging directory - A gathering point for items that are to be deployed. This is just a way station for modules and other jars. The artifacts are collected here so they can be copied all at once into the deploy directory in order to prevent reloading the application multiple times when multiple modules are being updated.
Deploy directory - The directory where the application is deployed and run.
The table below shows the commands used to create, modify and delete the various directories. Since various commands depend on others, there are generally several commands that will affect a given directory. We list here only the commands that would be most commonly used or useful.
One of the key things to note here is the cleanBuild command removes the entire build directory, requiring all of the build steps to be run again. This is generally not what you want or need to do since Gradle's up-to-date checks should be able to determine when things need to be rebuilt. (If that does not seem to be the case, please file a bug.) The one exception to this rule is when we change the LabKey version number after a release. Then you need to do a cleanBuild to get rid of the artifacts with the previous release version in their names.
Application Build Steps
Source code is built into jar files (and other types of files) in a module's build directory.
The result is a .module file, which contains potentially several jar files as well as other resources used by the module. This .module file is copied from the module's build directory into the staging directory and from there into the deploy directory. This is all usually accomplished with the './gradlew deployApp' command. The 'deployApp' task also configures and copies the application.properties file to the deploy directory.
Module Build Steps
A single module is deployed using the './gradlew deployModule' command. This command will create a .module file in the module's build directory, then copy it first into the staging modules directory and then into the deploy modules directory.
Build Tasks
A few important tasks:
Gradle Taskooooooooooooooooooooooooooooooooo
Description
gradlew tasks
Lists all of the available tasks in the current project.
gradlew pickPg
or
gradlew pickMSSQL
Use the pickPg task to configure your system by merging the settings in your pg.properties file with the application.properties template. This task is run the first time you build LabKey, and again any time you modify properties. See below.
If you are using Microsoft SQL Server use the task pickMSSQL and file mssql.properties instead.
gradlew deployApp
Build the LabKey Server source for development purposes. This is a development-only build that skips many important steps needed for production environments, including GWT compilation for popular browsers, gzipping of scripts, production of Java & JavaScript API documentation, and copying of important resources to the deployment location. Builds produced by this command will not run in production mode.
gradlew :server:modules:wiki:deployModule or server/modules/wiki>gradlew deployModule
For convenience, every module can be deployed separately. If your changes are restricted to a single module then building just that module is a faster option than a full build. Example, to build the wiki module: 'gradlew :server:modules:wiki:deployModule'.
gradlew deployApp -PdeployMode=prod
Build the LabKey Server source for deployment to a production server. This build takes longer than 'gradlew deployApp' but results in artifacts that are suitable and optimized for production environments.
gradlew cleanBuild
Delete all artifacts from previous builds. This should be used sparingly as it requires Gradle to start all over again and not capitalize on the work it's done before.
gradlew stopTomcat
Can be used to stop the server if IntelliJ crashes or can't shutdown LabKey for some reason.
gradlew projects
Lists the current modules included in the build.
gradlew :server:test:uiTest
Open the test runner's graphical UI.
gradlew :server:test:uiTest -Psuite=DRT
Run the basic automated test suite.
Gradle tasks can also be invoked from within IntelliJ via the "Gradle projects" panel, but this has not been widely tested.
pickPg
Use the pickPg task to configure your system before you run the server the first time, and any time you change either of the component files later. This task takes the contents of two files:
If you are using Microsoft SQL Server use the task pickMSSQL and file mssql.properties instead.
Parallel Build Feature
To improve performance, the LabKey build is configured by default to use Gradle's parallel build feature. This is controlled by these properties in the gradle.properties file:
org.gradle.parallel=true org.gradle.workers.max=3
By changing these properties, you can turn off parallel builds or adjust the number of threads Gradle uses.If Gradle warns that "JVM heap space is exhausted", add more memory as described in the topic Gradle Tips and Tricks.
Exclude Specific Gradle Tasks
If you want to build without completing all build tasks, such as without overwriting the application.properties file, you can use "-x" syntax to exclude a task. Find out specific tasks using:
Instead of building all of your modules from source, you can choose to use prebuilt module archives that will be downloaded from an artifact server.
Which modules are built from source is controlled by the buildFromSource property. This property can be set at the <LK_ENLISTMENT level> (i.e. the root of your enlistment), at the project level, at the global (user) level, or on the command line (with overrides happening in the order given).
The default properties file in <LK_ENLISTMENT level>/gradle.properties has the following property defined
buildFromSource=true
This setting will cause a build command to construct all the jars and the .module files that are necessary for a server build. But if you are not changing code, you do not need to build everything from source. The following table illustrates how you can specify the properties to build just the source code you need to build.
If you want to...
then...
Build nothing from source
Set "buildFromSource=false" in one of:
Command line (-PbuildFromSource=false)
<HOME_DIR>/.gradle/gradle.properties
<LK_ENLISTMENT level>/gradle.properties
Then run
gradlew deployApp
Build everything from source
Set "buildFromSource=true" in one of
Command line (-PbuildFromSource=true)
<HOME_DIR>/.gradle/gradle.properties
<LK_ENLISTMENT level>/gradle.properties
Then run
gradlew deployApp
Build a single module from source
Set "buildFromSource=false" in
<LK_ENLISTMENT level>/gradle.properties
Then EITHER (most efficient) run:
deployModule command for that module (e.g., gradlew :server:opt:cds:deployModule)
OR (Less efficient)
create a gradle.properties file within the directory of the module you want to build from source
include the setting buildFromSource=true
issue the deployApp command
Build everything from source EXCEPT one or more modules
Set "buildFromSource=true" in
<LK_ENLISTMENT level>/gradle.properties
Then:
create a gradle.properties file within the directory of each module you don't want to build from source
include the setting buildFromSource=false
issue the deployApp command
Build a subset of modules from source
Set "buildFromSource=false" in
<LK_ENLISTMENT level>/gradle.properties
Then EITHER (most efficient) run:
deployModule command for the modules you wish to build (e.g., gradlew :server:opt:cds:deployModule)
OR (Less efficient):
create a gradle.properties file within the module directories of each module you want to build from source
include the setting buildFromSource=true in each file
The LabKey Server module build process is designed to be flexible, consistent, and customizable. The process is driven by a manifest file that dictates which module directories to build. Module directories are listed either individually or using wildcards.A few of the options this enables:
Modify your build manifest files to remove modules that you never use, speeding up your build.
Add your custom module directories to an existing build location (e.g., /server/modules) to automatically include them in the standard build.
By default, the standard build tasks use the manifest file "/settings.gradle". You can edit this file to customize the modules that are built. settings.gradle includes a mix of wildcards and individually listed modules.Common syntax includes variations on lists of "includedModules" and "excludedModules".
Wildcard Example
The following builds every module under the localModules directory that is contained in an app directory. Note that app is the parent of the module directories, not the module directory itself:
def excludedModules = ["inProgress"] // The line below includes all modules under the localModules directory that are contained in a directory "app". // The modules are subdirectories of app, not app itself. BuildUtils.includeModules(this.settings, rootDir, [**/localModules/**/app/*], excludedModules);
Module Directory Example
The following builds every module directory in "server/modules", except those listed as excludedModules:
def excludedModules = ["inProgress"] // The line below includes all modules in the server/modules directory (except the ones indicated as to be excluded) BuildUtils.includeModules(this.settings, rootDir, [BuildUtils.SERVER_MODULES_DIR], excludedModules);
Individual Module Example
The following adds the 'dataintegration' module to the build:
// The line below is an example of how to include a single module include ":server:modules:dataintegration"
Custom Module Manifests
You can also create custom module manifest files. For example, the following manifest file 'local_settings.gradle' provides a list of individually named modules:
local_settings.gradle
include ':server:bootstrap' include ':server:modules:commonAssays:ms2' include ':server:modules:commonAssays:luminex' include ':server:modules:platform:announcements' include ':server:modules:platform:api' include ':server:modules:platform:assay' include ':server:modules:platform:audit' include ':server:modules:platform:core' include ':server:modules:platform:devtools' include ':server:modules:platform:experiment' include ':server:modules:platform:filecontent' include ':server:modules:platform:internal' include ':server:modules:platform:issues' include ':server:modules:platform:list' include ':server:modules:platform:mothership' include ':server:modules:platform:pipeline' include ':server:modules:platform:query' include ':server:modules:platform:search' include ':server:modules:platform:study' include ':server:modules:platform:wiki' include ':server:modules:targetedms' include ':server:testAutomation:modules:dumbster' include ':server:testAutomation:modules:pipelinetest'
The following uses the custom manifest file in the build:
gradlew -c local_settings.gradle deployApp
gradle/settings files
Instead of supplying a local settings file and using the -c option, you can put your setting file in the <LK_ENLISTMENT>/gradle/settings directory and use the moduleSet property to tell Gradle to pick up this settings file. The property value should be the basename of the settings file you wish to you. For example:
gradlew -PmoduleSet=all
...will cause Gradle to incorporate the file <LK_ENLISTMENT>/gradle/settings/all.gradle to define the set of projects (modules) to include.
Skip a Module
You may choose to exclude certain modules from your Gradle configuration via the property excludedModules, introduced with Gradle plugins version 1.43.0. This property is handy if you want to include a set of modules defined via a moduleSet but need to exclude a select one or two, or even a full directory of modules. The value for this property should be a comma-separated string, where each element of that string can be one of the following
a directory name (e.g., 'commonAssays')
an unambiguous name for the enlistment directory of the module (e.g., 'flow')
the full gradle path to the module (e.g., ':server:modules:commonAssays:flow').
For example, the following command will exclude all the modules in the commonAssays directory:
This property can also be set in a gradle.properties file (for example, in your home directory's .gradle/gradle.properties file) in case you know you always want to exclude configuring and building some modules that are included in an enlistment.
excludedModules=:server:modules:commonAssays:nab
The excludeModules property is applied in the settings file, so it causes the project for the module to be excluded from gradle's configuration entirely.The skipBuild property is deprecated and does not work well if you also are declaring dependencies in the module you wish to skip building. Prefer the excludedModules property instead.
The build targets can be made to ignore a module if you define the property skipBuild for this project. You can do this by adding a gradle.properties file in the project's directory with the following content:
skipBuild
Note that we check only for the presence of this property, not its value.The skipBuild property will cause the project to be included by gradle, so it would show up if you ran the "gradle projects" command, but it will remain largely not configured.
Undeploy a Module
When you have been building with a module and want to remove it completely, you should add it to an "excludedModules" list in your settings.gradle file and also need to use the unDeploy task in the module's directory.For example, to remove the "dumbster" module you would add it to either "excludedModules" or "excludedExternalModules" (depending on where it is located in your enlistment) and before rebuilding, go to where it is installed and execute:
The build process will download the versions of npm and node that are specified by the properties npmVersion and nodeVersion, respectively, which are defined in the root-level gradle.properties file of the LabKey enlistment. This means you no longer need to, nor should you, install npm and node yourself.To build LabKey, you must have the build-prod script defined. See below.
Node and npm Versions
If you are curious about the current versions in use, you can find those details in this file. Look for 'npmVersion' and 'nodeVersion':
You can also use direnv to ensure that the node and npm versions you are using match what the LabKey build is using.
package.json Scripts
To build LabKey, the package.json file (at the root of your module) should contain specific "scripts" targets, namely:
Required:
build-prod: Script used by npm to build in production mode.
Used for Local Development:
build: Script used by npm to build in dev mode.
start: Run locally to start the webpack dev server
test: Run locally to run jest tests.
clean: Convenience script to clean generated files. Does not delete downloaded node_modules.
Required for TeamCity (also requires TeamCity configuration to set up modules to run jest tests):
teamcity: Runs jest tests with a TeamCity plugin
Running npm on Command Line
If you run npm commands from the command line, it is strongly recommended that you put the downloaded versions of node and npm in your path. (Both are required to be in your path for the npm commands to run.) For these purposes, on non-Windows systems, the deployApp command creates symlinks for both node and npm in the directory .node at the root level of your enlistment:
trunk$ ls -la .node/ total 0 drwxr-xr-x 4 developer staff 128 Jun 14 14:11 . drwxr-xr-x 38 developer staff 1216 Jun 18 10:36 .. lrwxr-xr-x 1 developer staff 90 Jun 14 14:11 node -> /Users/developer/Development/labkey/trunk/build/modules/core/.node/node-v8.11.3-darwin-x64 lrwxr-xr-x 1 developer staff 77 Jun 14 13:58 npm -> /Users/developer/Development/labkey/trunk/build/modules/core/.node/npm-v5.6.0
You should add <LK_ENLISTMENT>/.node/node/bin and <LK_ENLISTMENT>/.node/npm/bin to your path environment variable. This is a one-time operation. The target of these links will be updated to reflect any version number changes. (Note that the directory won't exist until you run a deployApp command.)For Windows systems, these links can be created manually using the MKLINK command, which must be run as a user with permissions to create symbolic links. These manually created links will not be updated when version numbers change, so it may be just as easy to forego the symbolic link and alter your path to include the build/modules/core/.node/node* and build/modules/core/.node/npm* directories (the directory name will include the version number of the tool).Confirm that node.js has been installed by calling
When you build with Gradle, it creates a .gradle file in the directory in which the build command is issued. This directory contains various operational files and directories for Gradle and should not be checked in.To tell git to ignore these files and directories, add this in the .gitignore file for your Git module repositories:
.gradle
We’ve also updated the credits page functionality for the Gradle build and the build now produces a file dependencies.txt as a companion to the jars.txt file in a module’s resources/credits directory. This is not a file that needs to be checked in, so it should also be ignored, by including this in the .gitignore file for Git module repositories:
resources/config/dependencies.txt
Or, if your Git repository contains multiple modules:
Gradle will check by default for new versions of artifacts in the repository each time you run a command. If you are working offline or for some other reason don’t want to check for updated artifacts, you can set Gradle in offline mode using the --offline flag on the command line. If you don’t want to have to remember this flag, you can either set up an alias or use an init script to set the parameter startParameter.offline=trueIf you are running commands from within IntelliJ, there is also a setting for turning on and off offline mode. Toggle this setting in the Gradle window as shown in this screenshot:
Gradle is generally very good about keeping track of when things have changed and so you can, and should, get out of the habit of wiping things clean and starting from scratch because it just takes more time. If you find that there is some part of the process that does not recognize when its inputs have changed or its outputs are up-to-date, please file a bug or open a ticket on your support portal so we can get that corrected.The gradle tasks also provide much more granularity in cleaning. Generally, for each task that produces an artifact, we try to have a corresponding cleaning task that removes that artifact. This leads to a plethora of cleaning tasks, but there are only a few that you will probably ever want to use.In this topic we summarize the most commonly useful cleaning tasks, indicating what the outcome of each task is and providing examples of when you might use each.
Running 'gradlew cleanDeploy' removes the build/deploy directory. This will also stop the tomcat server if it is running.Result:
Stops Tomcat
Removes the staging directory: <LK_ENLISTMENT>/build/staging
Removes the deploy directory: <LK_ENLISTMENT>/build/deploy
Use when:
Removing a set of modules from your LabKey instance (i.e., after updating settings.gradle to remove some previously deployed modules)
Troubleshooting a LabKey server that appears to have built properly
cleanBuild
Running 'gradlew cleanBuild' removes the build directory entirely, requiring all of the build steps to be run again. This will also stop the tomcat server if it is running. This is the big hammer that you should avoid using unless there seems to be no other way out.This is generally not what you want or need to do since Gradle's up-to-date checks should be able to determine when things need to be rebuilt. The one exception to this rule is that when the LabKey version number is incremented with each monthly release, you need to do a cleanBuild to get rid of all artifacts with the previous release version in their names.Result:
Stops Tomcat
Removes the build directory: <LK_ENLISTMENT>/build
Use when:
Updating LabKey version number in the gradle.properties file in an enlistment
All else fails
Module Cleaning
The most important tasks for cleaning modules follow. The example module name used here is "MyModule".
:server:modules:myModule:clean
Removes the build directory for the module. This task comes from the standard Gradle lifecycle, and is generally followed by a deployModule or deployApp command.Result:
Note that this will have little to no effect on a running server instance. It will simply cause gradle to forget about all the building it has previously done so the next time it will start from scratch.
Use when:
Updating dependencies for a module
Troubleshooting problems building a module
:server:modules:myModule:undeployModule
Removes all artifacts for this module from the staging and deploy directories. This is the opposite of deployModule. deployModule copies artifacts from the build directories into the staging (<LK_ENLISTMENT>/build/staging) and then the deployment (<LK_ENLISTMENT>/build/deploy) directories, so undeployModule removes the artifacts for this module from the staging and deployment directories. This will cause a restart of a running server since Tomcat will recognize that the deployment directory is changed.Result:
Removes module's deploy directory and deployed .module file: <LK_ENLISTMENT>/build/deploy/modules/myModule.module and <LK_ENLISTMENT>/build/deploy/modules/myModule
Restarts Tomcat.
Use when:
There were problems at startup with one of the modules that you do not need in your LabKey server instance
Always use when switching between feature branches because the artifacts created in a feature branch will have the feature branch name in their version number and thus will look different from artifacts produced from a different branch. If you don't do the undeployModule, you'll likely end up with multiple versions of your .module file in the deploy directory and thus in the classpath, which will cause confusion.
:server:modules:myModule:reallyClean
Removes the build directory for the module as well as all artifacts for this module from the staging and deploy directories. Use this to remove all evidence of your having built a module. Combines undeployModule and clean to remove the build, staging and deployment directories for a module.Result:
Gradle properties can be set in four different places:
Globally - Global properties are applied at the user or system-level. The intent of global properties is the application of common settings across multiple Gradle projects. For example, using globally applied passwords makes maintenance easier and eliminates the need to copy passwords into multiple areas.
In a Project - Project properties apply to a whole Gradle project, such as the LabKey Server project. Use project-level properties to control the default version numbers of libraries and other resources, and to control how Gradle behaves over the whole project, for example, whether or not to build from source.
In a Module - Module properties apply to a single module in LabKey Server. For example, you can use module-level properties to control the version numbers of jars and other resources.
On the command line, for example: -PbuildFromSource=false
Property settings lower down this list override any settings made higher in the list. So a property set at the Project level will override the same property set at the Global level, and a property set on the command line will override the same property set at the Global, Project, or Module levels. See the Gradle documentation for more information on setting properties.
Global Properties
The global properties file should be located in your home directory:
On Windows: C:\Users\<you>\.gradle\gradle.properties
On Mac/Linux: /Users/<you>/.gradle/gradle.properties
A template global properties file is available at <LK_ENLISTMENT>/gradle/global_gradle.properties_template in your enlistment.
For instructions on setting up the global properties file, see Set Up a Development Machine.Some notable global properties are described below:
deployMode - Possible values are "dev" or "prod". This controls whether a full set of build artifacts is generated that can be deployed to production-mode servers, or a more minimal set that can be used on development mode machines.
systemProp.tomcat.home - Value is the path to the Tomcat home directory. Gradle needs to know the Tomcat home directory for some of the dependencies in the build process. IntelliJ does not pick up the $CATALINA_HOME environment variable, so if working with the IntelliJ IDE, you need to set the tomcat.home system property either here (i.e., as a global Gradle property) or on the command line with -Dtomcat.home=/path/to/tomcat/home. Regardless of OS, use the forward slash (/) as a file separator in the path (yes, even on Windows).
includeVcs - We do not check the value of this property, only its presence or absence. If present, the population of VCS revision number and URL is enabled in the module.properties file. Generally this is left absent when building in development mode and present when building in production mode.
artifactory_user - Your artifactory username (if applicable)
artifactory_password - The password for the above account, if applicable.
Project Root Properties
The project properties file resides at:
<LK_ENLISTMENT>/gradle.properties
For the most part, this file sets the version numbers for external tools and libraries, which should not be modified.Notable exceptions are described below:
buildFromSource - Indicates if we should use previously published artifacts or build from source. This setting applies to all projects unless overridden on the command line or in a project's own gradle.properties file. The default properties file in <LK_ENLISTMENT>/gradle.properties sets buildFromSource to "true". This setting will cause a build command to construct all the jars and the .module files that are necessary for a server build. If you are not changing code, consider setting "buildFromSource=false". The following table illustrates how you can buildFromSource to build just the source code you need to build.
If you want to...
...then...
Build nothing from source
Set buildFromSource=false and run "gradlew deployApp".
Build everything from source
Set buildFromSource=true and run "gradlew deployApp".
Build a single module from source
Set buildFromSource=false and run the deployModule command for that module (for example, "gradlew :server:modules:wiki:deployModule"). Alternatively, you could create a gradle.properties file within the module you want to build from source, include the setting "buildFromSource=true", and call "gradlew deployApp".
Build a subset of modules from source
Set buildFromSource=false and run the deployModule command for the subset of modules you wish to build. Alternatively, you could create gradle.properties files within the modules you want to build from source, include the setting "buildFromSource=true" in each, and call "gradlew deployApp".
Note that, unlike other boolean properties we use, the value of the buildFromSource property is important. This is because we want to be able to set this property for individual modules and the mere presence of a property at a higher level in the property hierarchy cannot be overridden at a lower level. If the property is defined, but not assigned a value, this will have the same effect has setting it to false.Community developers who want to utilize 'buildFromSource=false' will need to limit the list of modules built. Use a custom manifest of modules, such as 'local_settings.gradle', as described in the topic Customize the Build.
labkeyVersion - The default version for LabKey artifacts that are built or that we depend on. Override in an individual module's gradle.properties file as necessary.
Module-Specific Properties
This is most commonly used to set version number properties for the dependencies of a particular module, which, while not necessary, is a good practice to use for maintainability. An example module-specific properties file, from /server/test that sets various version properties:
Other common properties you may want to set at the module level:
skipBuild - This will cause Gradle to not build artifacts for this module even though the module's Gradle project is included by the settings.gradle file. We do not check the value of this property, only its presence.
buildFromSource - As mentioned above, this will determine whether a module should be built from source or whether its artifacts should be downloaded from the artifact repository. We do check the value of this property, so it should be set to either true or false. If the property is set but not assigned a value, this will have the same effect as setting it to false.
Gradle: How to Add Modules
Adding a Module: Basic Process
The basic workflow for building and deploying your own module within the LabKey source tree goes as follows:
Apply plugins in the module's build.gradle file.
Declare dependencies in build.gradle (if necessary).
Include the module in your build manifest file (if necessary).
Sync Gradle.
Deploy the module to the server.
Details for file-based modules vs. Java modules are described below.
Apply Plugins
File-based Modules
For file-based modules, add the following plugins to your build.gradle file:build.gradle
Or, as of gradlePluginsVersion 1.22.0 (and LabKey version 21.1):
plugins { id 'org.labkey.build.module' }
Stand-alone Modules
For modules whose source resides outside the LabKey Server source repository, you must provide your own gradle files, and other build resources, inside the module. The required files are:
settings.gradle - The build manifest where plugin repositories can be declared and project name is set
build.gradle - The build script where plugins are applied, tasks, are added, dependencies are declared, etc. .
module.template.xml - The build system uses this file in conjunction with module.properties to create a deployable properties file at /config/module.xml.
You will also probably want to put a gradle wrapper in your module. You can do this by using the gradle wrapper from the LabKey distribution with the following command in your module's directory:
/path/to/labkey/enlistment/gradlew wrapper
This will create the following in the directory where the command is run:
gradlew.bat - The Windows version of the Gradle wrapper. You get this when you install the Gradle wrapper.
gradle - directory containing the properties and jar file for the wrapper
If you do not have a LabKey enlistment or access to another gradle wrapper script, you will first need to install gradle in order to install the wrapper.The module directory will be laid out as follows. Note that this directory structure is based on the structure you get from createModule (except for module.template.xml, gradlew):
When your module code requires some other artifacts like a third-party jar file or another LabKey module, declare a dependency on that resource inside the build.gradle file. The example below adds a dependency to the workflow module. Notice that the dependency is declared for the 'apiJarFile' configuration of the LabKey module. This is in keeping with the architecture used for LabKey Modules.
Here we use the 'external' configuration instead of the implementation configuration to indicate that these libraries should be included in the lib directory of the .module file that is created for this project. The 'implementation' configuration extends this LabKey-defined 'external' configuration.There are many other examples in the server source code, for example:
With the default settings.gradle file in place, there will often be no changes required to incorporate a new module into the build. By placing the module in server/modules, it will automatically be picked up during the initialization phase. If you put the new module in a different location, you will need to modify the settings.gradle file to include this in the configuration.For example adding the following line to your build manifest file incorporates the helloworld module into the build. For details see Customize the Build.
include ':localModules:helloworld'
Sync Gradle
In IntelliJ, on the Gradle projects panel, click Refresh all gradle projects.
Deploy the Module
In your module's directory, call the following gradle task to build and deploy it:
If the module you are developing has dependencies on third-party libraries or modules other than server/platform/api or server/platform/core, you will need to add a build.gradle file in the module’s directory that declares these dependencies.
External dependencies should be declared using the LabKey-defined "external" configuration. This configuration is used when creating the .module file to know which libraries to include with the module. In the LabKey gradle plugins, we have a utility method (BuildUtils.addExternalDependency) for adding these dependencies that allows you to supply some extra metadata about the dependency related to its licensing and source as well as a short description. This metadata is then displayed on the Admin Console's Credits page. Using this method for external dependencies should be preferred over the usual dependency declaration syntax from Gradle. The arguments to BuildUtils.addExternalDependency are the current project and an ExternalDependency object. The constructor for this object takes 7 arguments as follows:
The artifact coordinates
The name of the component used by the module
The name of the source for the component
The URL for the source of the component
The name of the license
The URL for the license
A description of the purpose for this dependency
The proper artifact coordinates are needed for external dependencies to be found from the artifact repository. The coordinates consist of the group name (string before the first colon, e.g., 'org.apache.activemq'), the artifact name (string between the first and second colons, e.g., 'acitvemq-core'), and version number (string after the second colon and before the third colon, if any, e.g., '4.2.1'). You may also need to include a classifier for the dependency (the string after the third colon, e.g., 'embedded'). To find the proper syntax for the external dependencies, you can query MavenCentral, look for the version number you are interested in and then copy and paste from the gradle tab. It is generally best practice to set up properties for the versions in use.If the library you need is not available in one of the existing repositories, those who have access to the LabKey Artifactory can navigate to the "ext-release-local" artifact set, click the Deploy link in the upper right corner, and upload the JAR file. Artifactory will attempt to guess what a reasonable group, artifact, and version number might be, but correct as needed. Once added, it can be referenced in a module's build.gradle file like any other dependency. The artifact will need a proper pom file, so you should be sure to check the box next to "Generate Default POM/Deploy jar's Internal POM" when deploying.
Internal Dependencies
For internal dependencies, the BuildUtils.addLabKeyDependecy method referenced above will reference the buildFromSource gradle property to determine if a project dependency should be declared (meaning the artifacts are produced by a local build) or a package dependency (meaning the artifacts are pulled from the artifact server). The argument to this method is a map with the following entries:
project: The current project where dependencies are being declared
config: The configuration in which the dependency is being declared
depProjectPath: The (Gradle) path for the project that is the dependency
depProjectConfig : The configuration of the dependency that is relevant. This is optional and defaults to the default configuration or an artifact without a classifier
depVersion: The version of the dependency to retrieve. This is optional and defaults to the parent project’s version if the dependent project is not found in the gradle configuration or the dependent project’s version if it is found in the gradle configuration.
transitive: Boolean value to indicate if the dependency should be transitive or not
specialParams: This is a closure that can be used to configure the dependency, just as you can with the closure for a regular Gradle dependency.
To declare a compile-time dependency between one module and the API jar of a second module, you will do this:
If your module relies on the API jar file of another module, but does not need the module itself (likely because the code checks for the presence of the module
before using its functionality), you can use the "labkey" configuration when declaring this dependency. It is likely this dependency should not be transitive:
If the api jar file of the module you depend on exposes classes from a different jar file used to build this jar file, you should declare a dependency on the "runtimeElements" configuration from the other module. That module should be declaring its dependency to that separate jar file as an 'external' dependency. (This works because internally gradle's now-standard 'api' configuration, on which 'runtimeElements' is based, extends from the LabKey 'external' configuration.)
Module Dependencies
The moduleDependencies module-level property is used by LabKey server to determine the module initialization order and to control the order in which SQL scripts run. It is also used when publishing the .module file to record the proper dependencies in the pom file for that publication. These dependencies should be declared within a module's build.gradle file.For moduleA that depends on moduleB, you would add the following line to the moduleA/build.gradle file:
The behavior when a module is not in the settings.gradle file and/or when you do not have an enlistment for a module is as follows:
If :server:myModules:moduleB is not included in the settings.gradle file, moduleB will be treated like an external dependency and its .module file will be downloaded from Artifactory and placed in the build/deploy/modules directory by the deployApp command
If :server:myModules:moduleB is included in your settings.gradle file, but you do not have an enlistment in moduleB, by default, this will cause a build error such as "Cannot find project for :server:myModules:moduleB". You can change this default behavior by using the parameter -PdownloadLabKeyModules, and this will cause the .module file to be downloaded from Artifactory and deployed to build/deploy/modules, as in the previous case
If :server:myModules:moduleB is included in settings.gradle and you have an enlistment in moduleB, it will be built and deployed as you might expect.
As of gradlePlugin version 1.8, the .module files for LabKey Server are being published in a separate Maven group from the api jar files along with appropriate pom files that capture the module dependencies declared in their build.gradle files. The group for the .module files is org.labkey.module, while the group for the api jar files is org.labkey.api. This means that you can more easily pull in an appropriate set of modules to create a running LabKey server instance without building modules from source. In particular, if you add dependencies to the basic set of modules required for functional LabKey server within your module's build.gradle file, you should not need to enlist in the source for these modules or include them in your settings.gradle file. For example, within the build.gradle file for moduleA, you can declare the following dependencies
Without having an enlistment in any of the base modules (api, audit, core, etc.) or inclusion of the corresponding projects in your settings.gradle file, the deployApp task will pull in the modules that are required for a functional server. Note that the set of modules you have in your deployment will be a superset of the ones declared in the dependency closure, because of the dependencies declared within the modules' published pom files.There is also a utility method available in the BuildUtils class of the LabKey gradle plugins that can be used to declare the base module dependencies, so the above could be changed to
If you are building locally and want to include a .module file that is not an explicit dependency of any of your other modules, this can be done by declaring a dependency from the server project's modules configuration to the other module you wish to deploy. This is probably easiest to do by adding something like the following in your module's build.gradle file:
Adding Premium Modules to Developer Machines (Premium Feature)
Any developers working with Premium Editions of LabKey Server may want to include one or more premium modules in their local development machines. As an example, in order to use ETLs, you must have the dataintegration module on your server. Premium deployments include this module, but individual developers may also need it to be present on their development machines.To include this module in your build, include a dependency to 'pull' the prebuilt module from the LabKey Artifactory.Step 1: Access to the LabKey Artifactory is required and can be provided upon request to your Account Manager. Once access has been granted, you have a few options for providing your credentials when building. Learn more in this topic:
Step 2: Once you have the necessary access, declare a dependency on the desired module in the build.gradle file for a module that is present in your source enlistment.
If you are developing your own module, you can use its build.gradle file. This is the preferred option.
If you don't have a custom module, you can use the server/platform/list module's build.gradle file.
However, note that if any of the modules you are adding depend on the "list" module themselves (such as some biologics modules), use /server/platform/issues instead.
When you build locally, the prebuilt dataintegration module will be included and thus available on your local server.
Troubleshooting
If you include this dependency but do not have the appropriate credentials on Artifactory, you'll see a build error similar to the following. Note that this can happen when your credentials have not been granted as well as if they have been locked for some reason. Contact your Account Manager for assistance.
Could not resolve [library name]… > Could not GET 'https://labkey.jfrog.io/artifactory/...rest of URL ending in.pom'. Received status code 403 from server:
Resolve Conflicts
After adding a new external dependency, or updating the version of an existing external dependency, you will want to make sure the dependency hasn't introduced a version inconsistency with other modules. To do this, you can run the task 'showDiscrepancies' and you will want to include as many modules as possible for this task, so using the module set 'all' is a good idea:
./gradlew -PmoduleSet=all showDiscrepancies
If there are any discrepancies in external jar version numbers, this task will produce a report that shows the various versions in use and by which modules as shown here.
commons-collections:commons-collections has 3 versions as follows: 3.2 [:server:modules:query, :server:modules:saml] 3.2.1 [:externalModules:labModules:LDK] 3.2.2 [:server:api]
Each of these conflicts should be resolved before the new dependency is added or updated. Preferably, the resolution will be achieved by choosing a different version of a direct dependency in one or more modules. The task 'allDepInsight' can help to determine where a dependency comes from:
If updating direct dependency versions does not resolve the conflict, you can force a certain version of a dependency, which will apply to direct and transitive dependencies. See the root-level build.gradle file for examples of the syntax for forcing a version.
Version Conflicts in Local Builds
When version numbers are updated, either for LabKey itself or for external dependencies, a local build can accumulate multiple, conflicting versions of certain
jar files in its deploy directory, or the individual module build directories. This is never desirable. With gradlePlugin version 1.3, tasks have been added
to the regular build process that check for such conflicts.By default, the build will fail if a version conflict is detected, but the property 'versionConflictAction'
can be used to control that behavior. Valid values for this property are:
'delete' - this causes individual files in the deploy directory that are found in conflict with ones that are to be produced by the build to be deleted.
> Task :server:api:checkModuleJarVersions INFO: Artifact versioning problem(s) in directory /labkeyEnlistment/build/modules/api/explodedModule/lib: Conflicting version of commons-compress jar file (1.14 in directory vs. 1.16.1 from build). Conflicting version of objenesis jar file (1.0 in directory vs. 2.6 from build). INFO: Removing existing files that conflict with those from the build. Deleting /labkeyEnlistment/build/modules/api/explodedModule/lib/commons-compress-1.14.jar Deleting /labkeyEnlistment/build/modules/api/explodedModule/lib/objenesis-1.0.jar
BUILD SUCCESSFUL in 5s
Note that when multiple versions of a jar file are found to already exist in the build directory, none will be deleted. Manual intervention is required here to choose which version to keep and which to delete.
Execution failed for task ':server:api:checkModuleJarVersions'. > Artifact versioning problem(s) in directory /labkeyEnlistment/build/modules/api/explodedModule/lib: Multiple existing annotations jar files. Run the :server:api:clean task to remove existing artifacts in that directory.
* Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
'fail' (default) - this causes the build to fail when the first version conflict or duplicate version is detected.
Execution failed for task ':server:api:checkModuleJarVersions'. > Artifact versioning problem(s) in directory /labkeyEnlistment/build/modules/api/explodedModule/lib: Conflicting version of commons-compress jar file (1.14 in directory vs. 1.16.1 from build). Conflicting version of objenesis jar file (1.0 in directory vs. 2.6 from build). Run the :server:api:clean task to remove existing artifacts in that directory.
BUILD FAILED in 20s
'warn' - this will issue a warning message about conflicts, but the build will succeed. This can be useful in finding how many conflicts you have since the 'fail' option will show only the first conflict that is found.
> Task :server:api:checkModuleJarVersions WARNING: Artifact versioning problem(s) in directory /labkeyEnlistment/build/modules/api/explodedModule/lib: Conflicting version of commons-compress jar file (1.14 in directory vs. 1.16.1 from build). Conflicting version of objenesis jar file (1.0 in directory vs. 2.6 from build). Run the :server:api:clean task to remove existing artifacts in that directory.
BUILD SUCCESSFUL in 5s
Though these tasks are included as part of the task dependency chains for building and deploying modules, the four tasks can also be executed individually, which
can be helpful for resolving version conflicts without resorting to cleaning out the entire build directory. The tasks are:
checkModuleVersions - checks for conflicts in module file versions
checkWebInfLibJarVersions - checks for conflicts in jar files included in the WEB-INF/lib directory
checkModuleJarVersions - checks for conflicts in the jar files included in an individual module
checkVersionConflicts - runs all of the above tasks
Use gradlew -h to see the various options available for running gradle commands.By default, gradle outputs information related to the progress in building and which tasks it considers as up to date or skipped. If you don't want to see this, or any other output about progress of the build, add the -q flag:
gradlew -q projects
Set up an alias if you're a command-line kind of person who can't abide output to the screen.
Offline Mode
If working offline, you will want to use the --offline option to prevent it from contacting the artifact server. Note that you won't have success if you do this for your first build.Or you can toggle offline mode in IntelliJ, as shown in the following image:
Efficient Builds
If doing development in a single module, there is a command available to you that can be sort of a one-stop shopping experience:
/path/to/gradlew deployModule
This will build the jar files, and the .module file and then copy it to the build/deploy/modules directory, which will cause Tomcat to refresh.
Proteomics Binaries - Download or Skip
A package of proteomics binaries is available on Artifactory for your platform. When you build from source, they can be downloaded by Gradle and deployed to the build/deploy/bin directory of your local installation. Once downloaded, they won't be fetched again unless you remove the build/deploy/bin directory, such as by doing a cleanBuild.
Downloading Proteomics Binaries
If you are working with the MS2 module, you may want to opt in to downloading these binaries with the property "includeProteomicsBinaries" when building:
For convenience, you can also define the includeProteomicsBinaries property in your <user>/.gradle/gradle.properties file.Note: Previously, you had to opt out of downloading these binaries ("excludeProteomicsBinaries"). This is now the default behavior, though you may want to leave the old property defined if you develop against older LabKey releases.
Restoring Proteomics Binaries
If you exclude them and later wish to restore these binaries, first do a 'cleanBuild', then when you next 'deployApp' (with the includeProteomicsBinaries property on the command line or in your gradle.properties file) they will be downloaded.
Pre-Built Proteomics Binaries
Prebuilt binaries are available for manual download and deployment to your pipeline tools directory, or to your system PATH:
Build tasks are available at a fairly granular level, which allows you to use just the tasks you need for your particular development process. The targeting is done by providing the Gradle path (Gradle paths use colons as separators instead of slashes in either direction) to the project as part of the target name. For example, to deploy just the wiki module, you would do the following from the root of the LabKey enlistment:
./gradlew :server:modules:wiki:deployModule
Note that Gradle requires only unique prefixes in the path elements, so you could achieve the same results with less typing as follows:
./gradlew :se:m:w:depl
And when you mistype a target name it will suggest possible targets nearby. For example, if you type
gradlew pick_pg
Gradle responds with:
* What went wrong:
Task 'pick_pg' not found in project ':server'. Some candidates are: 'pickPg'.
Gradle's Helpful Tasks
Gradle provides many helpful tasks to advertise the capabilities and settings within the build system. Start with this and see where it leads you
gradlew tasks
Other very useful tasks for troubleshooting are
projects - lists the set of projects included in the build
dependencies - lists all dependencies, including transitive dependencies, for all the configurations for your build
Paths and Aliases
Placing the gradlew script in your path is not ideal in an environment where different directories/branches may use different versions of gradle (as you will invariably do if you develop on trunk and release branches). Instead, you can:
Always run the gradlew command from the root of the enlistment using ./gradlew
You may also want to create aliases for your favorite commands in Gradle.
Performance
There are various things you can do to improve performance:
Use targeted build steps to do just what you know needs to be done.
Use the -a option on the command line to not rebuild project dependencies (though you should not be surprised if sometimes your changes do require a rebuild of dependencies that you're not aware of and be prepared to remove that -a option).
Increase the number of parallel threads in use. This is done by setting the property org.gradle.workers.max in the root-level gradle.properties file. By default, this is set to 3. If you do not set this property, Gradle will attempt to determine the optimal number of threads to use. This may speed things up, but can also be memory-intensive, so you may need to increase the memory on the JVM gralde uses.
You can add more memory to the JVM by setting the org.gradle.jvmargs property to something like this:
org.gradle.jvmargs=-Xmx4g
If you want to update your properties with either of these settings, We suggest that you put them in your <user>/.gradle/gradle.properties file so they apply to all gradle instances and so you won't accidentally check them in.
Distributions
If you don't need the absolute latest code, and would be happy with the latest binary distribution, do the following to avoid the compilation steps altogether.
Put it in a directory that does not contain another file with the same extension as the distribution (.tar.gz)
Use the ./gradlew :server:cleanStaging :server:cleanDeploy :server:deployDistribution command, perhaps supplying a -PdistDir=/path/to/dist/directory property value to specify the directory that contains the downloaded distribution file.
By default, running gradlew deployApp creates a development build. It creates the minimum number of build artifacts required to run LabKey Server on a development machine. Some of these artifacts aren't required to run LabKey Server (such as pre-creating a .gz version of resources like .js files so the web server doesn't need to dynamically compress files for faster download), and others can be used directly from the source directories when the server is run in development mode (via the -DdevMode=true JVM argument). This means the development builds are faster and smaller than they would otherwise be.Note that individual modules built in development mode will not deploy to a production server. On deployment, the server will show the error: "Module <module-name> was not compiled in production mode". You can correct this by running
or, to build an individual module in production mode, you can add the following line to the module.properties file:
BuildType: Production
Production servers do not have access to the source directories, and should be optimized by performance, so they require that all resources be packaged in each module's build artifacts. This can be created by running this instead:
If you have existing build artifacts on your system, you will need to run gradlew cleanBuild first so that the build recognizes that it can't use existing .module files. The --no-build-cache switch ensures that gradle doesn't pull pre-built dev mode artifacts from its cache.All standard LabKey Server distributions (the .tar.gz downloads) are compiled in production mode.
The general process of setting up and using a development machine is described in Set Up a Development Machine. This topic provides additional guidance to use when setting up an OSX machine for LabKey development.
Take note of the username/password you select during installation.
Linux:
Install PostgreSQL using your Linux machine's native tools or download a binary package. Follow the instructions in the downloaded package to install PostgreSQL.
5. Set up environment variables:
We use <LK_ENLISTMENT> to refer to the location where you placed your enlistment. Substitute it where shown.
Add this location to your PATH: <LK_ENLISTMENT>/build/deploy/bin
This directory won't exist yet, but add it to the PATH anyway. The Gradle build downloads binaries used by the server, such as graphvis, dot, and proteomics tools, to this directory.
You can do this via traditional linux methods (in ~/.bash_profile) or via OSX's plist environment system.To add the environment variables using ~/.zshrc or ~/.bash_profile (depending on your default shell), edit the file and add the lines, customizing for your own local paths to your enlistment and java:
The setup for IntelliJ is described in the common documentation. There may be differences for OSX, and version differences, including but not limited to:
The menu path for setting path variables is IntelliJ IDEA > Preferences > Appearance & Behavior > Path Variables.
This topic covers troubleshooting some problems you may encounter setting up a development machine and building LabKey Server from source. Roughly divided into categories, there is some overlap among sections, so searching this page may be the most helpful way to find what you need.
When encountering build errors when first starting a dev machine, double check that you have carefully followed the setup instructions. Check the output of this command to confirm that your distribution is set up correctly:
gradlew projects
Build errors after switching branches or updating can be due to a mismatch in module versions. You can see a report of all the versions of your modules by running:
gradlew gitBranches
Note that you should also confirm, using IntelliJ, that the root <LK_ENLISTMENT> location is also on the correct branch.
Linux Service Executable Name
If you are on Linux and see an error similar to this:
labkey_server.service - LabKey Server Application
Loaded: bad-setting (Reason: Unit labkey_server.service has a bad unit file setting.)
Active: inactive (dead)
<DATE><SERVER DETAILS> /etc/systemd/system/labkey_server.service:19: Neither a valid executable name>
It is likely because you need to provide the full path to Java at the head of the "ExecStart" line in the labkey_server.service file. The default we send shows a <JAVA_HOME> variable, and you need to update this line to use the full path to $JAVA_HOME/bin/java instead.
Properties Not Picked Up
In some cases, you may have customized the application.properties in the <LK_ENLISTMENT>/server/configs directory, such as to enable SSL, set an encryption key, or use a port other than 8080, etc. If you are running your server and don't see those changes being reflected, be sure that you ran the 'gradlew pickPg' task after changing the properties. That task copies and merges the files from the /server/configs location to where they are actually run. You can check this file to confirm your changes to either pg.properties or application.properties were picked up and will be used by the running server:
A dev machine sets up both a server.port and a shutdown management port. These two port values must be different, and there must not be anything else running on either port.
If you have other processes using either port, the errors in the log may not be specific to this issue.
Unable to Start Gradle Daemon
If you are unable to start any Gradle daemon (i.e. even "./gradlew properties" raises an error), check to confirm that you are using the 64-bit version of the JDK. If you accidentally try to use the 32-bit version, that VM has a max heap size of about 1.4GB, which is insufficient for building LabKey.Note that "java -version" will specify "64-bit" if it is, but will not specify "32-bit" for that (incorrect) version. Instead the version line reads "Client VM" where you need to see "64-bit". At present the correct result of "java -version" should be:
openjdk version "17.0.10" 2024-01-16 OpenJDK Runtime Environment Temurin-17.0.10+7 (build 17.0.10+7) OpenJDK 64-Bit Server VM Temurin-17.0.10+7 (build 17.0.10+7, mixed mode, sharing)
An error like this may indicate that you are using the 32 bit version, as it :
FAILURE: Build failed with an exception.
* What went wrong: Unable to start the daemon process. This problem might be caused by incorrect configuration of the daemon. For example, an unrecognized jvm option is used. For more details on the daemon, please refer to https://docs.gradle.org/8.6/userguide/gradle_daemon.html in the Gradle documentation. Process command line: C:labkeyappsjdk-17binjava.exe -XX:+UseParallelGC --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.prefs/java.util.prefs=ALL-UNNAMED --add-opens=java.base/java.nio.charset=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED -Xmx2048m -Dfile.encoding=windows-1252 -Duser.country=US -Duser.language=en -Duser.variant -cp C:Usersmohara.gradlewrapperdistsgradle-8.6-binafr5mpiioh2wthjmwnkmdsd5wgradle-8.6libgradle-launcher-8.6.jar -javaagent:C:Usersmohara.gradlewrapperdistsgradle-8.6-binafr5mpiioh2wthjmwnkmdsd5wgradle-8.6libagentsgradle-instrumentation-agent-8.6.jar org.gradle.launcher.daemon.bootstrap.GradleDaemon 8.6 Please read the following process output to find out more: ----------------------- Error occurred during initialization of VM Could not reserve enough space for 2097152KB object heap
Server Not Starting Using a LabKey-Provided Run/Debug Config
Problem: Server does not start using one of the LabKey-provided Run/Debug configurations. In the console window the following error appears:
/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/bin/java -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:59311,suspend=y,server=n -Dcatalina.base=./ -Dcatalina.home=./ -Djava.io.tmpdir=./temp -Ddevmode=true -ea -Dsun.io.useCanonCaches=false -Xmx1G -XX:MaxPermSize=160M -classpath "./bin/*;/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar" -javaagent:/Users/susanh/Library/Caches/IntelliJIdea2016.3/groovyHotSwap/gragent.jar -Dfile.encoding=UTF-8 org.apache.catalina.startup.Bootstrap start Connected to the target VM, address: '127.0.0.1:59311', transport: 'socket' Error: Could not find or load main class org.apache.catalina.startup.Bootstrap Disconnected from the target VM, address: '127.0.0.1:59311', transport: 'socket'
Cause: This is most likely caused by an incorrect path separator in the Run/Debug configuration's classpath argument.Solution: Edit the Run/Debug configuration and change the separator to the one appropriate to your platform (semicolon for Windows; colon for Mac/Linux).
Fatal Error in Java Runtime Environment
Error: When starting LabKey or importing data to a new server, you might see a virtual machine crash similar to this:
# # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x0000000000000000, pid=23893, tid=39779 # # JRE version: Java(TM) SE Runtime Environment (8.0_45-b14) (build 1.8.0_45-b14) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode bsd-amd64 compressed oops) # Problematic frame: # C 0x0000000000000000 # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
Cause: These are typically bugs in the Java Virtual Machine itself.Solution: Ensuring that you are on the latest patched release for your preferred Java version is best practice for avoiding these errors. If you have multiple versions of Java installed, be sure that JAVA_HOME and other configuration is pointing at the correct location. If you are running through the debugger in IntelliJ, check the JDK configuration. Under Project Structure > SDKs check the JDK home path and confirm it points to the newer version.
Exception in compressClientLibs task
If you encounter this error when building in production mode:
* What went wrong: Execution failed for task ':server:modules:platform:query:compressClientLibs'. > org.xml.sax.SAXException: java.lang.NullPointerException java.lang.NullPointerException
It may indicate a parsing error from yuicompressor which handles javascript and css minification. The problem is usually that the code you added is using
elements of ES6 (like the spread operator) that the parser doesn't support.Solution: Fall back to older javascript syntax.
Cannot find class X when hot-swapping
Cause: IntelliJ puts its build output files, which it uses when doing hot-swapping, in different directories than the command-line Gradle build does, so if you have not built the entire project (or at least the jars that the file you are attempting to swap in depends on) within IntelliJ, IntelliJ will not find them.Solution: Open the Gradle window in IntelliJ and run the root-level "build" task from that window.
IntelliJ Troubleshooting
IntelliJ Version
If you are using IntelliJ and experience errors building or upgrading LabKey from source, check to see if your IntelliJ IDEA is up to date. You can download the the latest version here:
For example, you might see an error like this when using gradle refresh from within intelliJ:
Build model 'org.jetbrains.plugins.gradle.model.GradleExtensions' for root project 'labkey'
IntelliJ Warnings and Errors
Warning: Class "org.apache.catalina.startup.Bootstrap" not found in module "LabKey": You may ignore this warning in the Run/Debug Configurations dialog in IntelliJ.
Error: Could not find or load main class org.apache.catalina.startup.Bootstrap on OSX (or Linux): you might see this error in the console when attempting to start LabKey server. Update the '-classpath' VM option for your Run/Debug configuration to have Unix (:) path separators, rather than Windows path separators (;).
On Windows, if you are seeing application errors, you can try resetting the winsock if it has gotten into a bad state. To reset:
Open a command window in administrator mode
Type into the command window: netsh winsock reset
When you hit enter, that should reset the winsock and your application error may be resolved. You might need to restart your computer for the changes to take effect.
IntelliJ Slow
You can help IntelliJ run faster by increasing the amount of memory allocated to it. The specific method may vary depending on your version of IntelliJ and OS. Learn more here:
Problem: After doing a Gradle Refresh, some of the classes with package names like org.labkey.SOME_SCOPE.xml (e.g., org.labkey.security.xml) are not found.Cause: This is usually due to the relevant schema jars not having been built yet.Solution: Run the deployApp command and then do a Gradle Refresh within IntelliJ. Though you can use a more targeted Gradle task schemaJar to build just the jar file that is missing, you might find yourself running many of these to resolve all missing classes, so it is likely most efficient to simply use deployApp
Gradle Troubleshooting
Understanding how gradle builds the server, and the relationships between building and cleaning can help you diagnose many build problems. See these related topics:
We maintain release notes for the gradlePlugins that describe changes and bug fixes to the plugins. Check these notes to find out if the problem you are having has already been addressed. Be sure to pay attention to the "Earliest compatible LabKey version" indicated with each release to know if it is compatible with your current LabKey version. If you want to update to a new version of the gradle plugins to pick up a change, you need only update the gradlePluginsVersion property in the root-level gradle.properties file:
gradlePluginsVersion=2.7.2
Could not Resolve All Files for Configuration
Problem: When compiling a module, an error such as the following appears
Execution failed for task ':server:modules:myModule:compileJava'. > Could not resolve all files for configuration ':server:modules:myModule:compileClasspath'. > Could not find XYZ-api.jar (project :server:modules:XYZ).
Similarly, after doing a Gradle Refresh, some of the files that should be on the classpath are not found within IntelliJ. If you look in the Build log window, you see messages such as this:
<ij_msg_gr>Project resolve errors<ij_msg_gr><ij_nav>/Development/labkeyEnlistment/build.gradle<ij_nav><i><b>root project 'labkeyEnlistment': Unable to resolve additional project configuration.</b><eol>Details: org.gradle.api.artifacts.ResolveException: Could not resolve all dependencies for configuration ':server:modules:myModule:compileClasspath'.<eol>Caused by: org.gradle.internal.resolve.ArtifactNotFoundException: Could not find XYZ-api.jar (project :server:modules:XYZ).</i>
Cause: The settings.gradle file has included only a subset of the dependencies within the transitive dependency closure of your project and Gradle is unable to resolve the dependency for the api jar file. In particular, though your settings file likely includes the :server:modules:XYZ project, it likely does not include another project that myModule depends on that also depends on XYZ. That is, you have the following dependencies:
myModule depends on otherModule
myModule depends on XYZ
otherModule depends on XYZ
And in your settings file you have:
include ':server:modules:myModule' include ':server:modules:XYZ'
(Notice that 'otherModule' is missing here.)
When gradle resolves the dependency for otherModule, it will pick it up from the artifact repository. That published artifact contains a reference to the API jar file for XYZ and Gradle will then try to resolve that dependency using your :server:modules:XYZ project, but referencing an artifact with a classifier (the -api suffix here) is not something currently supported by Gradle, so it will fail. This may cause IntelliJ (particularly older versions of IntelliJ) to mark some dependencies within this dependency tree as "Runtime" instead of "Compile" dependencies, which will cause it to not resolve some references to classes within the Java files.Solution: Include the missing project in your settings.gradle file
include ':server:modules:myModule' include ':server:modules:otherModule' include ':server:modules:XYZ'
Module Not Deployed
If you notice that your deployed server does not contain all the modules you expected, the first thing to verify is that your settings.gradle file includes all the modules you intend to build. You do this by running the following command
gradlew projects
If the project is listed as expected in this output, the next thing to check is that the project has the appropriate plugins applied. See the tip about plugins below for more information.
Module Plugins Not Applied or Missing Properties orMethods
* What went wrong: A problem occurred evaluating project ':server:modules:myModule'. > Could not find method implementation() for arguments com.sun.mail:jakarta.mail:1.6.5 on object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.
Cause: As of LabKey Server version 21.3.0 we no longer apply the LabKey gradle plugins for all modules under the server/modules directory by default. Each module is responsible for applying the plugins it requires. When running a deployApp command, if we find a gradle project that contains a module.properties file but does not have a task named 'module', a message like the following will be shown:
The following projects have a 'module.properties' file but no 'module' task. These modules will not be included in the deployed server. You should apply either the 'org.labkey.build.fileModule' or 'org.labkey.build.module' plugin in each project's 'build.gradle' file. :server:modules:myModule :server:modules:otherModule
Solution: Apply the appropriate plugin in the module's build.gradle file by including either
plugins { id 'org.labkey.build.fileModule' }
OR
plugins { id 'org.labkey.build.module' }
depending on whether the project houses a file-based module or a Java module, respectively. See the documentation on creating Gradle modules for more information.
Starting Over with Gradle + IntelliJ
If you get into a situation where the IntelliJ configuration has been corrupted or created in a way that is incompatible with what we expect (which can happen if, for example, you say 'Yes' when IntelliJ asks if you want to have it link an unlinked Gradle project), these are the steps you can follow to start over with a clean setup for Gradle + IntelliJ:
Shut down IntelliJ.
Revert the <LK_ENLISTMENT>/.idea/gradle.xml file to the version checked in to VCS.
Remove the <LK_ENLISTMENT>/.idea/modules directory.
Remove the <LK_ENLISTMENT>/.idea/modules.xml file.
Start up IntelliJ.
Open the Gradle window (View > Tool Windows > Gradle) and click the Refresh icon, as described above.
This may also be required if the version of LabKey has been updated in your current enlistment but you find that your IntelliJ project still refers to jar files created with a previous version.
Log4J
log4j2.xml Appender
Problem: The server seems to have started fine, but there are no log messages in the console after server start up.Cause: The log4j2.xml file that controls where messages are logged does not contain the appropriate appender to tell it to log to the console. This appender is added when you deploy the application in development mode (as a consequence of running the deployApp command).In log4j2, a custom appender is defined by creating a plugin. The log4j documentation has an explanation with an example for a custom appender here:
If LabKey fails to start successfully, check the steps above to ensure that you have configured your JDK and development environment correctly. Some common errors you may encounter include:org.postgresql.util.PSQLException: FATAL: password authentication failed for user "<username>" or java.sql.SQLException: Login failed for user "<username>"These error occurs when the database user name or password is incorrect. If you provided the wrong user name or password in the .properties file that you configured above, LabKey will not be able to connect to the database. Check that you can log into the database server with the credentials that you are providing in this file.java.net.BindException: Address already in use: JVM_Bind:<port x>:This error occurs when another instance of LabKey, Tomcat, or another application is running on the same port. Specifically, possible causes include:
LabKey is trying to double bind the specified port.
LabKey (or Tomcat) is already running (as a previous service, under IntelliJ, etc.
Microsoft Internet Information Services (IIS) is running on the same port.
Another application is running on the same port.
In any case, the solution is to ensure that your development instance of LabKey is running on a free port. You can do this in one of the following ways:
Shut down the instance of LabKey or the application that is running on the same port.
Change the port for the other instance or application.
Edit the application.properties file to specify a different port for your installation of LabKey.
It is also possible to have misconfigured LabKey to double bind a given port. Check to see if your application.properties file has these two properties set to the same value:
server.port context.httpPort
java.lang.NoClassDefFoundError: com/intellij/rt/execution/application/AppMain:
or Error: Could not find or load main class com.intellij.rt.execution.application.AppMain:In certain developer configurations, you will need to add an IntelliJ utility JAR file to your classpath.
Edit the Debug Configuration in IntelliJ.
Under the "VM Options" section, find the "-classpath" argument.
Find your IntelliJ installation. On Windows machines, this is typically "C:\Program Files\JetBrains\IntelliJ IDEA <Version Number>" or similar. On OSX, this is typically "/Applications/IntelliJ IDEA <Version Number>.app" or similar.
The required JAR file is in the IntelliJ installation directory, and is ./lib/idea_rt.jar. Add it to the -classpath argument value, separating it from the other values with a ":" on OSX and a ";" on Windows.
Save your edits and start LabKey.
Database State Troubleshooting
If you build the LabKey source yourself from the source tree, you may need to periodically delete and recreate your LabKey database. The daily drops often include SQL scripts that modify the data and schema of your database.
Database Passwords Not Working
If your password contains a backslash, it may not get copied correctly from your properties file (i.e. pg.properties) to application.properties during the build and deploy process. Either use a password without a backslash, or double-escape the backslash within the properties file: (e.g. '\\\\').
System PATH
LabKey Server will automatically load certain binaries and tools (graphviz, dot, proteomics binaries) from Artifactory and expects to find them on the user's PATH. Be sure to include the following location in your system PATH. This directory won't exist before you build your server, but add it to the path anyway.
<LK_ENLISTMENT>\build\deploy\bin
For example, C:\labkeyEnlistment\build\deploy\bin.
Environment Variables Troubleshooting
JAVA_TOOL_OPTIONS
Most users will not have this problem. However, if you see a build error something like the following:
error: unmappable character for encoding ASCII
then setting this environment variable may fix the problem
export JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8
Ext Libraries
Problem: Ext4 is not defined.
The page or a portion of the page is not rendering appropriately. Upon opening the browser console you see an error stating "Ext4 is not defined". This usually occurs due to a page not appropriately declaring client-side code dependencies.Example of the error as seen in Chrome console.
Solution
Declare a dependency on the code/libraries you make use of on your pages. Depending on the mechanism you use for implementing your page/webpart we provide different hooks to help ensure dependent code is available on the page.
File-based module view: It is recommended that you use file-scoped dependencies by declaring a view.xml and using the <dependencies> attribute.
Java: In Java you can use ClientDependency.fromPath(String p) to declare dependencies for your view. Note, be sure to declare these before the view is handed off to rendering otherwise your dependency will not be respected.
JSP: Override JspBase.addClientDependencies(ClientDependencies dependencies) in the .jsp. Here is an example.
Background: In the 17.3 release we were able to move away from ExtJS 4.x for rendering menus. This was the last “site-wide” dependency we had on ExtJS, however, we still continue to use it throughout the server in different views, reports, and dialogs. To manage these usages we use our dependency framework to ensure the correct resources are on the page. The framework provides a variety of mechanisms for module-based views, JavaServer Pages, and Java.
Renaming Files and Directories
Occasionally a developer decides a module or file (and possibly associated classes) should be renamed for clarity or readability.Most file/directory name changes are handled transparently: rename locally through Intellij, verify your build and that tests pass, and commit.The big exception to this is doing a case-only rename. (e.g., "myfile.java" -> "MyFile.java"). Because Windows file systems are case-insensitive, this causes major headaches. Don't do it as described above.
But what if I really need to do a case-only rename?
Do a renaming to only change casing (ex. Mymodule -> myModule) in two rename steps:
First rename to an intermediate name that changes more than the casing: (ex. Mymodule -> myModule2)
Commit that change.
Then rename to the target final name (ex. myModule2 -> myModule)
Commit again.
Push your commits.
Squash-merge to develop.
Be sure to verify the build after each step. Depending upon what you are renaming, you may also have to rename classes.
Running a server in development mode provides enhanced logging and enables some features and behavior useful in development and troubleshooting.
Do not run your production server in development mode. Some devmode features reduce performance or make other tradeoffs that make them inappropriate for general usage on production servers.
Module Editing is enabled. Note that the server must also have access to the source path for module editing to work. You can tell if your server can access the source for a module by viewing the Module Information on the Admin Console. Expand the module of interest and if the Source Path is inaccessible, it will be shown in red.
Performance may be slower due to the additional logging and lack of caching.
Validation of XML is performed which may uncover badly formed XML files, such as those with elements in the wrong order.
A variety of test options that aren't appropriate for production use will be available. A few examples:
An "expire passwords every five seconds" option for testing the user password reset experience.
An insecure "TestSSO" authentication provider.
The ability to test ETL filter strategies with a Set Range option.
Run Server in Development Mode
Do not run your production server in development mode, but you might set up a shared development server to support future work, or have each developer work on their own local dev machine.To check whether the server is running in devmode:
Go to > Site > Admin Console.
On the Server Information tab, scroll to the Runtime Information section.
Note that the "caching" JVM system property can also be used to control just the caching behavior, without all of the other devmode behaviors. To disallow the normal caching, perhaps because files are being updated directly on the file system, add to the same place where you enabled devmode:
-Dcaching=false
Turn Off Devmode
To turn off devmode:
First stop your server.
Locate the -Ddevmode=true flag whereever it is being set. The location of this setting may vary depending on your version and how it was deployed, but it is generally set with JVM options.
Remove the flag. You do not set it to be false, simply remove it from the startup options.
The LabKey client libraries provide secure, auditable, programmatic access to LabKey data and services.The purpose of the client APIs is to let developers and statisticians write scripts or programs in various programming languages to extend and customize LabKey Server. The specifics depend on the exact type of integration you hope to achieve. For example, you might:
Analyze and visualize data stored in LabKey in a statistical tool such as R or SAS
Perform routine, automated tasks in a programmatic way.
Query and manipulate data in a repeatable and consistent way.
Enable customized data visualizations or user interfaces for specific tasks that appear as part of the existing LabKey Server user interface.
Provide entirely new user interfaces (web-based or otherwise) that run apart from the LabKey web server, but interact with its data and services.
All APIs are executed within a user context with normal security and auditing applied. This means that applications run with the security level of the user who is currently logged in, which will limit what they can do based on permissions settings for the current user.Currently, LabKey supports working with the following programming languages/environments.
Note that some download links above may show you a JFrog login page. You do not need to log in, but can close the login page using the 'X' in the upper right and your download will complete.
LabKey's JavaScript client library makes it easy to write custom pages and utilities on LabKey Server. A few examples of ways you might use the JavaScript API to access and/or update data:
Add JavaScript to a LabKey HTML page to create a custom renderer for your data, transforming and presenting the data to match your vision.
Upload an externally-authored HTML page that uses rich UI elements such as editable grids, dynamically trees, and special purpose data entry controls for LabKey Server tables.
Create a series of HTML/JavaScript pages that provide a custom workflow packaged as a module.
JavaScript API
The JavaScript API has evolved and expanded to provide new functionality related to endpoints and Domain operations. The latest version of the TypeScript-based API is available here:
In addition, the original JS API can still be used for the operations not included in the newer version, including classes that concern the DOM and user interface. Some examples are:
To access any original API classes from a TypeScript client, add one of the following at the top of a .tsx file (just under the imports), then use the LABKEY namespace as usual.
In this step, you will create the user interface for collecting requests. A wiki containing JavaScript can be designed to accept user input and translate it into the information needed in the database. In this example, users specify the desired reagent, a desired quantity, and some user contact information, submitting requests with a form like the following:
Folders and Permissions
First create a separate folder where your target users have "insert" permissions. Creating a separate folder allows you to grant these expanded permissions only for the folder needed and not to any sensitive information. Further, insertion of data into the lists can then be carefully controlled and granted only through admin-designed forms.
Log in to your server and navigate to your "Tutorials" project. Create it if necessary.
If you don't already have a server to work on where you can create projects, start here.
If you don't know how to create projects and folders, review this topic.
Create a new subfolder named "Reagent Request Tutorial." Accept the default "collaboration" folder type.
On the User/Permissions page, click Finish and Configure Permissions.
Uncheck Inherit permissions from parent.
Next to Submitter, add the group "Site: All Site Users".
Remove "Guest" from the Reader role if present.
Click Save and Finish.
You will now be on the home page of your new "Reagent Request Tutorial" folder.
Import Lists
Our example reagent request application uses two lists. One records the available reagents, the other records the incoming requests. Below you import the lists in one pass, using a "list archive". (We've pre-populated these lists to simulate a system in active use.)
Copy and paste the HTML/JavaScript code block into the Source tab.
Click Save & Close.
To make a more convenient dashboard for users/requesters: enter Page Admin Mode and remove the Subfolders web part.
The page reagentRequest now displays the submission form, as shown at the top of this page.See a live example.
Notes on the source code
The following example code uses LABKEY.Query.selectRows and LABKEY.Query.insertRows to handle traffic with the server. For example code that uses Ext components, see LABKEY.ext.Store.View the source code in your application, or view similar source in the interactive example. Search for the items in orange text to observe any or all of the following:
Initialization. The init() function pre-populates the web form with several pieces of information about the user.
User Info. User information is provided by LABKEY.Security.currentUser API. Note that the user is allowed to edit some of the user information obtained through this API (their email address and name), but not their ID.
Dropdown. The dropdown options are extracted from the Reagent list. The LABKEY.Query.selectRows API is used to populate the dropdown with the contents of the Reagents list.
Data Submission. To insert requests into the Reagent Requests list, we use LABKEY.Query.insertRows. The form is validated before being submitted.
Asynchronous APIs. The success in LABKEY.Query.insertRows is used to move the user on to the next page only after all data has been submitted. The success function executes only after rows have been successfully inserted, which helps you deal with the asynchronous processing of HTTP requests.
Default onFailure function. In most cases, it is not necessary to explicitly include an onFailure function for APIs such as LABKEY.Query.insertRows. A default failure function is provided automatically; create one yourself if you wish a particular mode of failure other than the simple, default notification message.
Confirmation page dependency. Note that this source code requires that a page named "confirmation" exists before you can actually submit a request. Continue to the next step: Step 2: Confirmation Page to create this page.
Code
<div style="float: right;">
<input value='View Source' type='button' id='viewSource'><br/><br/>
</div><form name="ReagentReqForm">
<table cellspacing="0" cellpadding="5" border="0">
<tr>
<td colspan="2">Please use the form below to order a reagent.
All starred fields are required.</td>
</tr>
<tr><td colspan="2"><br/></td></tr>
<tr>
<td colspan="2"><div id="errorTxt" style="display:none;color:red"></div></td>
</tr>
<tr>
<td valign="top" width="100"><strong>Name:*</strong></td>
<td valign="top"><input type="text" name="DisplayName" size="30"></td>
</tr>
<tr><td colspan="2" style="line-height:.6em"><br/></td></tr>
<tr>
<td valign="top" width="100"><strong>Email:*</strong></td>
<td valign="top"><input type="text" name="Email" size="30"></td>
</tr>
<tr><td colspan="2" style="line-height:.6em"><br/></td></tr>
<tr>
<td valign="top" width="100"><strong>UserID:*</strong></td>
<td valign="top"><input type="text" name="UserID" readonly="readonly" size="30"></td>
</tr>
<tr><td colspan="2" style="line-height:.6em"><br/></td></tr>
<tr>
<td valign="top" width="100"><strong>Reagent:*</strong></td>
<td valign="top">
<div>
<select id="Reagent" name="Reagent">
<option>Loading...</option>
</select>
</div>
</td>
</tr>
<tr><td colspan="2" style="line-height:.6em"><br/></td></tr>
<tr>
<td valign="top" width="100"><strong>Quantity:*</strong></td>
<td valign="top"><select id="Quantity" name="Quantity">
<option value="1">1</option>
<option value="2">2</option>
<option value="3">3</option>
<option value="4">4</option>
<option value="5">5</option>
<option value="6">6</option>
<option value="7">7</option>
<option value="8">8</option>
<option value="9">9</option>
<option value="10">10</option>
</select></td>
</tr>
<tr><td colspan="2" style="line-height:.6em"><br/></td></tr>
<tr>
<td valign="top" width="100"><strong>Comments:</strong></td>
<td valign="top"><textarea cols="53" rows="5" name="Comments"></textarea></td>
</tr>
<tr><td colspan="2" style="line-height:.6em"><br/></td></tr> <tr>
<td valign="top" colspan="2">
<div align="center">
<input value='Submit' type='button' id='submitIt'>
</td>
</tr>
</table>
</form>
<script type="text/javascript" nonce="<%=scriptNonce%>"> window.onload = init(); LABKEY.Utils.onReady(function() {
document.getElementById("viewSource")['onclick'] = gotoSource;
document.getElementById("submitIt")['onclick'] = submitRequest;
}); // Demonstrates simple use of LABKEY.ActionURL to show source for this page:
function gotoSource() {
window.location = LABKEY.ActionURL.buildURL("wiki", "source", LABKEY.ActionURL.getContainer(), {name: 'reagentRequest'});
} // Initialize the form by populating the Reagent drop-down list and
// entering data associated with the current user.
function init() {
LABKEY.Query.selectRows({
schemaName: 'lists',
queryName: 'Reagents',
success: populateReagents
}); document.getElementById("Reagent").selectedIndex = 0; // Set the form values
var reagentForm = document.getElementsByName("ReagentReqForm")[0];
reagentForm.DisplayName.value = LABKEY.Security.currentUser.displayName;
reagentForm.Email.value = LABKEY.Security.currentUser.email;
reagentForm.UserID.value = LABKEY.Security.currentUser.id;
} // Populate the Reagent drop-down menu with the results of
// the call to LABKEY.Query.selectRows.
function populateReagents(data) {
var el = document.getElementById("Reagent");
el.options[0].text = "<Select Reagent>";
for (var i = 0; i < data.rows.length; i++) {
var opt = document.createElement("option");
opt.text = data.rows[i].Reagent;
opt.value = data.rows[i].Reagent;
el.options[el.options.length] = opt;
}
} // Enter form data into the reagent request list after validating data
// and determining the current date.
function submitRequest() {
// Make sure the form contains valid data
if (!checkForm()) {
return;
} // Insert form data into the list.
LABKEY.Query.insertRows({
schemaName: 'lists',
queryName: 'Reagent Requests',
rowDataArray: [{
"Name": document.ReagentReqForm.DisplayName.value,
"Email": document.ReagentReqForm.Email.value,
"UserID": document.ReagentReqForm.UserID.value,
"Reagent": document.ReagentReqForm.Reagent.value,
"Quantity": parseInt(document.ReagentReqForm.Quantity.value),
"Date": new Date(),
"Comments": document.ReagentReqForm.Comments.value,
"Fulfilled": 'false'
}],
success: function(data) {
// The set of URL parameters.
var params = {
"name": 'confirmation', // The destination wiki page. The name of this parameter is not arbitrary.
"userid": LABKEY.Security.currentUser.id // The name of this parameter is arbitrary.
}; // This changes the page after building the URL. Note that the wiki page destination name is set in params.
var wikiURL = LABKEY.ActionURL.buildURL("wiki", "page", LABKEY.ActionURL.getContainer(), params);
window.location = wikiURL;
}
});
} // Check to make sure that the form contains valid data. If not,
// display an error message above the form listing the fields that need to be populated.
function checkForm() {
var result = true;
var ob = document.ReagentReqForm.DisplayName;
var err = document.getElementById("errorTxt");
err.innerHTML = '';
if (ob.value == '') {
err.innerHTML += "Name is required.";
result = false;
}
ob = document.ReagentReqForm.Email;
if (ob.value == '') {
if(err.innerHTML != '')
err.innerHTML += "<br>";
err.innerHTML += "Email is required.";
result = false;
}
ob = document.ReagentReqForm.Reagent;
if (ob.value == '') {
if(err.innerHTML != '<Select Reagent>')
err.innerHTML += "<br>";
err.innerHTML += "Reagent is required.";
result = false;
}
if(!result)
document.getElementById("errorTxt").style.display = "block";
return result;
}</script>
Now that you have created a way for users to submit requests, you are ready to create the confirmation page. This page will display the count of requests made by the current user, followed by a grid view of requests submitted by all users, similar to the following. You can sort and filter the grid to see specific subsets of requests, such as your own.After you have submitted one or more "requests" the tallies will change and your new requests will be shown.See a live example.
Create the Confirmation Page
Return to your "Reagent Request Tutorial" folder if you navigated away.
Click the (triangle) menu on Reagent Request Form and select New.
Name: "confirmation" (this page name is already embedded in the code for the request page, and is case sensitive).
Title: "Reagent Request Confirmation".
Confirm that the Source tab is selected.
Copy and paste the contents of the code section below into the source panel.
Click Save & Close.
You will see a grid displayed showing the sample requests from our list archive. If you already tried submitting one before creating this page, you would have seen an error that the confirmation page didn't exist, but you will see the request listed here.
Click Reagent Request Form in the Pages web part to return to the first wiki you created and submit some sample requests to add data to the table.
Notes on the JavaScript Source
LABKEY.Query.executeSql is used to calculate total reagent requests and total quantities of reagents for the current user and for all users. These totals are output to text on the page to provide the user with some idea of the length of the queue for reagents.Note: The length property (e.g., data.rows.length) is used to calculate the number of rows in the data table returned by LABKEY.Query.executeSql. It is used instead of the rowCount property because rowCount returns only the number of rows that appear in one page of a long dataset, not the total number of rows on all pages.
Code
<p>Thank you for your request. It has been added to the request queue and will be filled promptly.</p>
<div id="totalRequests"></div>
<div id="allRequestsDiv"></div>
<div id="queryDiv1"></div><script type="text/javascript" nonce="<%=scriptNonce%>"> window.onload = init(); function init() { var qwp1 = new LABKEY.QueryWebPart({
renderTo: 'queryDiv1',
title: 'Reagent Requests',
schemaName: 'lists',
queryName: 'Reagent Requests',
buttonBarPosition: 'top',
// Uncomment below to filter the query to the current user's requests.
// filters: [ LABKEY.Filter.create('UserID', LABKEY.Security.currentUser.id)],
sort: '-Date'
}); // Extract a table of UserID, TotalRequests and TotalQuantity from Reagent Requests list.
LABKEY.Query.executeSql({
schemaName: 'lists',
queryName: 'Reagent Requests',
sql: 'SELECT "Reagent Requests".UserID AS UserID, ' +
'Count("Reagent Requests".UserID) AS TotalRequests, ' +
'Sum("Reagent Requests".Quantity) AS TotalQuantity ' +
'FROM "Reagent Requests" Group BY "Reagent Requests".UserID',
success: writeTotals
}); } // Use the data object returned by a successful call to LABKEY.Query.executeSQL to
// display total requests and total quantities in-line in text on the page.
function writeTotals(data)
{
var rows = data.rows; // Find overall totals for all user requests and quantities by summing
// these columns in the sql data table.
var totalRequests = 0;
var totalQuantity = 0;
for(var i = 0; i < rows.length; i++) {
totalRequests += rows[i].TotalRequests;
totalQuantity += rows[i].TotalQuantity;
} // Find the individual user's total requests and quantities by looking
// up the user's id in the sql data table and reading off the data in the row.
var userTotalRequests = 0;
var userTotalQuantity = 0;
for(i = 0; i < rows.length; i++) {
if (rows[i].UserID === LABKEY.Security.currentUser.id){
userTotalRequests = rows[i].TotalRequests;
userTotalQuantity = rows[i].TotalQuantity;
break;
}
} document.getElementById('totalRequests').innerHTML = '<p>You have requested <strong>' +
userTotalQuantity + '</strong> individual bottles of reagents, for a total of <strong>'
+ userTotalRequests + '</strong> separate requests pending. </p><p> We are currently '
+ 'processing orders from all users for <strong>' + totalQuantity
+ '</strong> separate bottles, for a total of <strong>' + totalRequests
+ '</strong> requests.</p>';
}</script>
To further explore more possibilities available with JavaScript, let's add an R histogram (data visualization plot) of the "Reagent Requests" list to the confirmation page. This will create a page that looks like the following screencap.This is an optional step. If you wish you can skip to the last step in the tutorial: Step 4: Summary Report For Managers
Set Up R
If you have not already configured your server to use R, follow these instructions before continuing: Install and Set Up R.
Create an R Histogram
Return to the home page of your "Reagent Request Tutorial" folder by clicking the Reagent Request Tutorial link or using the project and folder menu.
Select > Manage Lists.
In the Available Lists grid, click Reagent Requests.
Paste the following code onto the Source tab (replacing the default contents).
if(length(labkey.data$userid) > 0){ png(filename="${imgout:histogram}") hist(labkey.data$quantity, xlab = c("Quantity Requested", labkey.url.params$displayName), ylab = "Count", col="lightgreen", main= NULL) dev.off() } else { write("No requests are available for display.", file = "${txtout:histogram}") }
Check the "Make this report available to all users" checkbox.
Scroll down and click Save.
Enter the Report Name: "Reagent Histogram". We assume this value in the wiki page to display it.
Click OK.
Click the Report tab to see the R report.
This histogram gives a view of all requests listed in the "Reagent Requests" table.
Update the Confirmation Page
Return to the main page by clicking the Reagent Request Tutorial link near the top of the page.
Open the "Reagent Request Confirmation" wiki page for editing. Click it in the Pages web part, then click Edit. We will make three changes to customize the page:
Add the following to the end of the block of <div> tags at the top of the page:
<div id="reportDiv">Loading...</div>
This will give users an indication that additional information is coming while the histogram is loading.
Uncomment the line marked with "// Uncomment below to filter the query to the current user's requests."
This will reduce the grid displayed; if you have not entered any sample requests, an empty table will be shown.
Add the following to the init() function after the "//Extract a table..." section:
// Draw a histogram of the user's requests. var reportWebPartRenderer = new LABKEY.WebPart({ partName: 'Report', renderTo: 'reportDiv', frame: 'title', partConfig: { title: 'Reagent Request Histogram', reportName: 'Reagent Histogram', showSection: 'histogram' } }); reportWebPartRenderer.render();
Click Save & Close.
You will now see the histogram on the Reagent Request Confirmation page.
If you see the error message "Unable to display the specified report," go back and confirm that the name of the report matches the name provided in the wiki content.
Link to a live example.Note that the R histogram script returns data for all users. The wiki page passes the report view of the dataset to the R script (via the partConfig parameter of LABKEY.WebPart). To see the web part configuration parameters available, see: Web Part Configuration Properties.When creating a filter over the dataset, you will need to determine the appropriate filter parameter names (e.g., 'query.UserID~eq'). To do so, go to the dataset and click on the column headers to create filters that match the filters you wish to pass to this API. Read the filter parameters off of the URL.You can pass arbitrary parameters to the R script by adding additional fields to partConfig. For example, you could pass a parameter called myParameter with a value of 5 by adding the line "myParameter: 5,". Within the R script editor, you can extract URL parameters using the labkey.url.params variable, as described at the bottom of the "Help" tab.
In this tutorial step you'll use more queries and JavaScript to create a report page for application managers, handy information that they can use to help coordinate their efforts to fulfill the requests.
The page will look like the following:See a live example.
Create Custom SQL Queries
First, create three custom SQL queries over the "Reagent Requests" list in order to distill the data in ways that are useful to reagent managers. Here, we create custom SQL queries using the LabKey UI, then use LABKEY.QueryWebPart to display the results as a grid. As part of writing custom SQL, we can add Metadata XML to provide a URL link to the subset of the data listed in each column.
Query #1: Reagent View
First define a query that returns all the reagents, the number of requests made, and the number requested of each.
Return to the home page of your "Reagent Request Tutorial" folder.
Select > Go To Module > Query.
Select the lists schema.
Click Create New Query.
Define your first of three SQL queries:
What do you want to call the new query?: Enter "Reagent View"
Which query/table do you want this new query to be based on?: Select Reagent Requests
Click Create and Edit Source.
Paste this SQL onto the Source tab (replace the default SQL query text):
SELECT "Reagent Requests".Reagent AS Reagent, Count("Reagent Requests".UserID) AS TotalRequests, Sum("Reagent Requests".Quantity) AS TotalQuantity FROM "Reagent Requests" Group BY "Reagent Requests".Reagent
Click the XML Metadata tab and paste the following (replace the default):
Depending on what requests have been entered, the results might look something like this:
Query #2: User View
The next query added will return the number of requests made by each user.
Click lists Schema above the grid to return to the Schema Browser. (Notice your new "Reagent View" request is now included.)
Click Create New Query.
Call this query "User View".
Again base it on Reagent Requests. Note that by default, it would be based on the last query you created, so change that selection before continuing.
Click Create and Edit Source.
Paste this into the source tab:
SELECT "Reagent Requests".Name AS Name, "Reagent Requests".Email AS Email, "Reagent Requests".UserID AS UserID, Count("Reagent Requests".UserID) AS TotalRequests, Sum("Reagent Requests".Quantity) AS TotalQuantity FROM "Reagent Requests" Group BY "Reagent Requests".UserID, "Reagent Requests".Name, "Reagent Requests".Email
Name the query "Recently Submitted" and again base it on the list Reagent Requests.
Click Create and Edit Source.
Paste this into the source tab:
SELECT Y."Name", MAX(Y.Today) AS Today, MAX(Y.Yesterday) AS Yesterday, MAX(Y.Day3) AS Day3, MAX(Y.Day4) AS Day4, MAX(Y.Day5) AS Day5, MAX(Y.Day6) AS Day6, MAX(Y.Day7) AS Day7, MAX(Y.Day8) AS Day8, MAX(Y.Day9) AS Day9, MAX(Y.Today) + MAX(Y.Yesterday) + MAX(Y.Day3) + MAX(Y.Day4) + MAX(Y.Day5) + MAX(Y.Day6) + MAX(Y.Day7) + MAX(Y.Day8) + MAX(Y.Day9) AS Total FROM (SELECT X."Name", CASE WHEN X.DayIndex = DAYOFYEAR(NOW()) THEN X.C ELSE 0 END AS Today, CASE WHEN X.DayIndex = DAYOFYEAR(NOW()) - 1 THEN X.C ELSE 0 END AS Yesterday, CASE WHEN X.DayIndex = DAYOFYEAR(NOW()) - 2 THEN X.C ELSE 0 END AS Day3, CASE WHEN X.DayIndex = DAYOFYEAR(NOW()) - 3 THEN X.C ELSE 0 END AS Day4, CASE WHEN X.DayIndex = DAYOFYEAR(NOW()) - 4 THEN X.C ELSE 0 END AS Day5, CASE WHEN X.DayIndex = DAYOFYEAR(NOW()) - 5 THEN X.C ELSE 0 END AS Day6, CASE WHEN X.DayIndex = DAYOFYEAR(NOW()) - 6 THEN X.C ELSE 0 END AS Day7, CASE WHEN X.DayIndex = DAYOFYEAR(NOW()) - 7 THEN X.C ELSE 0 END AS Day8, CASE WHEN X.DayIndex = DAYOFYEAR(NOW()) - 8 THEN X.C ELSE 0 END AS Day9, CASE WHEN X.DayIndex = DAYOFYEAR(NOW()) - 9 THEN X.C ELSE 0 END AS Day10 FROM ( SELECT Count("Reagent Requests".Key) AS C, DAYOFYEAR("Reagent Requests".Date) AS DayIndex, "Reagent Requests"."Name" FROM "Reagent Requests" WHERE timestampdiff('SQL_TSI_DAY', "Reagent Requests".Date, NOW()) < 10 GROUP BY "Reagent Requests"."Name", DAYOFYEAR("Reagent Requests".Date) ) X GROUP BY X."Name", X.C, X.DayIndex) Y GROUP BY Y."Name"
There is nothing to paste into the XML Metadata tab.
Click Save & Finish.
If you do not see much data displayed by the "Recently Submitted" query, the dates of reagent requests may be too far in the past. To see more data here, you can:
Manually edit the dates in the list to occur within the last 10 days.
Edit the source XLS to bump the dates to occur within the last 10 days, and re-import the list.
Create a bunch of recent requests using the reagent request form.
Create Summary Report Wiki Page
Click Reagent Request Tutorial to return to the main page.
On the Pages web part, select (triangle) > New to create a new wiki.
Copy and paste the code block into the Source tab.
Click Save & Close.
This summary page, like other grid views of data, is live - if you enter new requests, then return to this page, they will be immediately included.
Notes on the JavaScript Source
You can reopen your new page for editing or view the source code below to observe the following parts of the JavaScript API.
Check User Credentials
The script uses the LABKEY.Security.getGroupsForCurrentUser API to determine whether the current user has sufficient credentials to view the page's content.
Display Custom Queries
Use the LABKEY.QueryWebPart API to display our custom SQL queries in the page. Note the use of aggregates to provide sums and counts for the columns of our queries.
Display All Data
Lastly, to display a grid view of the entire "Reagent Requests" list on the page, use the LABKEY.QueryWebPart API, allowing the user to select and create views using the buttons above the grid.
// Ensure that the current user has sufficient permissions to view this page. LABKEY.Security.getGroupsForCurrentUser({ successCallback: evaluateCredentials });
// Check the group membership of the current user. // Display page data if the user is a member of the appropriate group. function evaluateCredentials(results) { // Determine whether the user is a member of "All Site Users" group. var isMember = false; for (var i = 0; i < results.groups.length; i++) { if (results.groups[i].name == "All Site Users") { isMember = true; break; } }
// If the user is not a member of the appropriate group, // display alternative text. if (!isMember) { var elem = document.getElementById("errorTxt"); elem.innerHTML = '<p>You do ' + 'not have sufficient permissions to view this page. Please log in to view the page.</p>' + '<p>To register for a labkey.org account, please go <a href="https://www.labkey.com/download-community-edition/">here</a></p>'; elem.style.display = "inline"; } else { displayData(); } }
// Display page data now that the user's membership in the appropriate group // has been confirmed. function displayData() { // Link to the Reagent Request list itself. LABKEY.Query.getQueryDetails({ schemaName: 'lists', queryName: 'Reagent Requests', success: function(data) { var el = document.getElementById("listLink"); if (data && data.viewDataUrl) { var html = '<p>To see an editable list of all requests, click '; html += '<a href="' + data.viewDataUrl + '">here</a>'; html += '.</p>'; el.innerHTML = html; } } });
// Display a summary of reagents var reagentSummaryWebPart = new LABKEY.QueryWebPart({ renderTo: 'reagentDiv', title: 'Reagent Summary', schemaName: 'lists', queryName: 'Reagent View', buttonBarPosition: 'none', aggregates: [ {column: 'Reagent', type: LABKEY.AggregateTypes.COUNT}, {column: 'TotalRequests', type: LABKEY.AggregateTypes.SUM}, {column: 'TotalQuantity', type: LABKEY.AggregateTypes.SUM} ] });
// Display a summary of users var userSummaryWebPart = new LABKEY.QueryWebPart({ renderTo: 'userDiv', title: 'User Summary', schemaName: 'lists', queryName: 'User View', buttonBarPosition: 'none', aggregates: [ {column: 'UserID', type: LABKEY.AggregateTypes.COUNT}, {column: 'TotalRequests', type: LABKEY.AggregateTypes.SUM}, {column: 'TotalQuantity', type: LABKEY.AggregateTypes.SUM}] });
// Display how many requests have been submitted by which users // over the past 10 days. var resolvedWebPart = new LABKEY.QueryWebPart({ renderTo: 'recentlySubmittedDiv', title: 'Recently Submitted', schemaName: 'lists', queryName: 'Recently Submitted', buttonBarPosition: 'none', aggregates: [ {column: 'Today', type: LABKEY.AggregateTypes.SUM}, {column: 'Yesterday', type: LABKEY.AggregateTypes.SUM}, {column: 'Day3', type: LABKEY.AggregateTypes.SUM}, {column: 'Day4', type: LABKEY.AggregateTypes.SUM}, {column: 'Day5', type: LABKEY.AggregateTypes.SUM}, {column: 'Day6', type: LABKEY.AggregateTypes.SUM}, {column: 'Day7', type: LABKEY.AggregateTypes.SUM}, {column: 'Day8', type: LABKEY.AggregateTypes.SUM}, {column: 'Day9', type: LABKEY.AggregateTypes.SUM}, {column: 'Total', type: LABKEY.AggregateTypes.SUM} ] });
Congratulations! You have created a functioning JavaScript application. Return to your tutorial page, make a few requests and check how the confirmation and summary pages are updated.
Converting a JavaScript application into a module has a number of advantages. For example, the application source can be checked into a source control environment, and it can be distributed and deployed as a module unit.The jstutorial.module file shows how to convert two of the application pages (reagentRequest and confirmation) into views within a module. The .module file is a renamed .zip archive. To unzip the file and see the source, rename it to "jstutorial.zip", and unzip it.To deploy and use the .module file:
This tutorial covers how to pass parameters between pages via a URL, then use those received URL parameters to filter a grid and customize reports. These features are helpful when building applications to make it easier for your users to navigate your data from a custom interface.
To accomplish this, you will:
1. Collect user input from an initial page
2. Build a parameterized URL to pass the user's input to a second page.
3. Use information packaged in the URL to filter a data grid.
We will use a list of reagents as our sample data, our finished application will filter the list of reagents to those that start with the user provided value. For example, if the user enters 'tri', the grid will display only those reagents whose name starts with 'tri'.See a completed version of what you will build in this tutorial.
Requirements
To complete this tutorial, you will need:
Admin or Developer permissions on a LabKey Server installation. If you don't already have this, see Set Up for Tutorials: Non-Trial for options.
In this tutorial step, we create a page to collect a parameter from the user. This value will be used to filter for items in the data that start with the text provided. For example, if the user enters 'tri', the server will filter for data records that start with the value 'tri'.
Set Up
First, set up the folder with underlying data to filter.
function buttonHandler() { if (document.SubmitForm.searchText.value) { //Set the name of the destination wiki page, //and the text we'll use for filtering. var params = {}; params['name']= 'showFilter'; params['searchText'] = document.SubmitForm.searchText.value; //alert(params['searchText']);
// Build the URL to the destination page. // In building the URL for the "Show Filtered Grid" page, we use the following arguments: // controller - The current controller (wiki) // action - The wiki controller's "page" action // containerPath - The current container // parameters - The parameter array we just created above (params) window.location = LABKEY.ActionURL.buildURL( "wiki", "page", LABKEY.ActionURL.getContainer(), params); } else { alert('You must enter a value to submit.'); } }
We use the "params" object to package up all the URL parameters. In this tutorial, we place only two parameters into the object, but you could easily add additional parameters of your choice. The two parameters:
name -- The name of the destination wiki page, with the value "showFilter". This page doesn't exist yet.
searchText -- The text we'll use for filtering on the "showFilter" page. This will be provided through user input.
Use the Wiki to Build the URL
In the Choose Parameters section, enter some text, for example, "a", and click Submit.
The destination page (showFilter) doesn't exist yet, so disregard any error.
Notice the URL in the browser which was built from the parameters provided, especially the query string portion following the '?': '?name=showFilter&searchText=a'.
// We use the 'searchText' parameter contained in the URL to create a filter. var myFilters = []; if (LABKEY.ActionURL.getParameter('searchText')) { var myFilters = [ LABKEY.Filter.create('Reagent', LABKEY.ActionURL.getParameter('searchText'), LABKEY.Filter.Types.STARTS_WITH) ] }
// In order to display the filtered list, we render a QueryWebPart // that uses the 'myFilters' array (created above) as its filter. // Note that it is recommended to either use the 'renderTo' config option // (as shown below) or the 'render( renderTo )' method, but not both. // These both issue a request to the server, so it is only necessary to call one. var qwp = new LABKEY.QueryWebPart({ schemaName : 'lists', queryName : 'Reagents', // Change to use a different list, ex: 'Instruments' renderTo : 'filteredTable', filters : myFilters });
}; </script> <div id="filteredTable"></div>
Notice that the entire list of reagents is displayed because no filter has been applied yet. The query string in the URL is currently "?name=showFilter".
Display a Filtered Grid
Now we are ready to use our parameterized URL to filter the data.
Click URL Tutorial to return to the main page.
In the Choose Parameters web part, enter some search text, for example the single letter 'a' and click Submit.
The URL is constructed and takes you to the destination page.
Notice that only those reagents that start with 'a' are shown.
Notice the query string in the URL is now "?name=showFilter&searchText=a".You can return to the Change Parameters web part or simply change the URL directly to see different results. For example, change the searchText value from 'a' to 'tri' to see all of the reagents that begin with 'tri'. The "showFilter" page understands the "searchText" value whether it is provided directly or via handoff from the other wiki page.
Finished
Congratulations! You've now completed the tutorial and created a simple application to pass user input via the URL to filter data grids.For another API tutorial, try Tutorial: Create Applications with the JavaScript API.
Once you have created a chart and filtered and refined it using the data grid and user interface, you can export it as JavaScript. Then, provided you have Developer permissions (are a member of the "Developer" site group), you can insert it into an HTML page, such as a wiki, and directly edit it. The powerful LabKey visualization libraries include many ways to customize the chart beyond features available in the UI. This lets you rapidly prototype and collaborate with others to get the precise presentation of data you would like.The exported JavaScript from a chart will:
Load the dependencies necessary for visualization libraries
Load the data to back the chart
Render the chart
Because the exported script selects data in the database directly, if the data changes after you export and edit, the chart will reflect the data changes as well.In this tutorial, we will:
This example uses the sample study datasets imported in the study tutorial. If you have not already set that up, follow the instructions in this topic: Install the Sample Study.
This tutorial starts by making a time chart grouped by treatment group, then exporting the JavaScript to use in the next tutorial steps. This example uses the sample study datasets imported as part of our tutorial example study.
Navigate to the home page of your sample study, "HIV Study." If you don't have one already, see Install the Example Study.
Click the Clinical and Assay Data tab.
Open the LabResults data set.
Select (Charts) > Create Chart.
Click Time.
For the X Axis select Visit-Based.
Drag CD4 from the column list to the Y Axis box.
Click Apply.
You will see a basic time chart. Before exporting the chart to Javascript, we can customize it within the wizard.
Click Chart Layout, then change the Subject Selection to "Participant Groups". Leave the default "Show Mean" checkbox checked.
Change the Number of Charts to "One per Group".
Click Apply.
In the Filters > Groups panel on the left, select Cohorts and deselect anything that was checked by default. The chart will now be displayed as a series of four individual charts in a scrollable window, one for each treatment group:
Export to JavaScript
Hover over the chart to reveal the Export buttons, and click to Export as Script.
You will see a popup window containing the HTML for the chart, including the JavaScript code.
Select All within the popup window and Copy the contents to your browser clipboard.
Click Close in the popup. Then Save your chart with the name of your choice.
Before proceeding, paste the copied chart script to a text file on your local machine for safekeeping. In this tutorial, we use the name "ChartJS.txt".
You can embed an exported JavaScript chart into a Wiki or any other HTML page without needing to make any modifications to what is exported. To complete this step you must have either "Platform Developer" or "Trusted Analyst" permissions.This topic is a continuation of the JavaScript Visualizations tutorial and begins with you holding the exported script on your clipboard or in a text file as described in the previous step.
Click the Overview tab to go to the home page of your study, or navigate to any tab where you would like to place this exported chart.
If the folder already contains a wiki page named "default", the new web part will display it. Choose New from the web part (triangle) menu.
Otherwise, click Create a new wiki page in the new wiki web part.
Give the page the name of your choice. Wiki page names must be unique, so be careful not to overwrite something else unintentionally.
Enter a Title such as "Draft of Chart".
Click the Source tab. Note: if there is no Source tab, click Convert To..., select HTML and click Convert.
Paste the JavaScript code you copied earlier onto the source tab. Retrieve it from your text file, "ChartJS.txt" if it is no longer on your browser clipboard.
Scroll up and click Save.
You could also add additional HTML to the page before or after the pasted JavaScript of the chart, or make edits as we will explore in the next tutorial step.
Caution: Do not switch to the Visual tab. The visual editor does not support this JavaScript element, so switching to that tab would cause the chart to be deleted. You will be warned if you click the Visual tab. If you do accidentally lose the chart, you can to recover the JavaScript using the History of the wiki page, your ChartJS.txt file, or by exporting it again from the saved timechart.
Scroll up and click Save & Close.
Return to the tab where you placed the new wiki web part. If it does not already show your chart, select Customize from the (triangle) menu for the web part and change the Page to display to the name of the wiki you just created. Click Submit.
Notice that the web part now contains the series of single timecharts as created in the wizard.
The chart wizard itself offers a variety of tools for customizing your chart. However, by editing the exported JavaScript for the chart directly you can have much finer grained control as well as make modifications that are not provided by the wizard.
Open your wiki for editing by clicking Edit or the pencil icon if visible.
Confirm that the Source tab is selected. Reminder: Do not switch to the Visual tab.
Scroll down to find the line that looks like this:
Replace that line with the following code block. It is good practice to mark your additions with comments such as those shown here.
// ** BEGIN MY CODE ** // create an accordion layout panel for each of the treatment group plots var accordionPanel = Ext4.create('Ext.panel.Panel', { renderTo: 'exportedChart', title: 'Time Chart: CD4 Levels per Treatment Group', width: 760, height: 500, layout: 'accordion', items: [] });
// loop through the array of treatment groups var groupIndex = 0; Ext4.each(chartConfig.subject.groups, function(group) { // add a new panel to the accordion layout for the given group var divId = 'TimeChart' + groupIndex ; accordionPanel.add(Ext4.create('Ext.panel.Panel', { title: group.label, html: '<div id="' + divId + '"></div>' }));
// clone and modify the queryConfig and chartConfig for the plot specific to this group var groupQueryConfig = Ext4.clone(queryConfig); groupQueryConfig.defaultSingleChartHeight = 350; groupQueryConfig.defaultWidth = 750; var groupChartConfig = Ext4.clone(chartConfig); groupChartConfig.subject.groups = [group];
// call the plot render method using the cloned config objects LABKEY.vis.TimeChartHelper.renderChartSVG(divId , groupQueryConfig, groupChartConfig);
groupIndex++; }); // ** END MY CODE **
Click Save and Close to view your new chart, which is now an "accordion panel" style. Click the and buttons on the right to expand/collapse the individual chart panels.
To embed an exported chart without surrounding user interface, create a simple file based module where your chart is included in a simple myChart.html file. Create a myChart.view.html file next to that page with the following content. This will load the necessary dependencies and create a page displaying only the simple chart. (To learn how to create a simple module, see Tutorial: Hello World Module.)
Note that API calls are case sensitive. Making a call like LabKey.query.selectRows({ ... will not work. You need to capitalize the word LABKEY as in LABKEY.Query.selectRows({ ...
Premium Resources AvailableSubscribers to Premium Editions of LabKey Server can find more examples in these topics:
Displays a query in the home/ProjectX folder. The containerFilter property being set to "allFolders" broadens the scope of the query to pull data from the entire site.
// Displays the named file set 'store1'. var wp1 = new LABKEY.WebPart({ title: 'File Store #1', partName: 'Files', partConfig: {fileSet: 'store1'}, renderTo: 'fileDiv' }); wp1.render();
</script>
Insert a Wiki Web Part
Note that the Web Part Configuration Properties covers the configuration properties that can be set for various types of web parts inserted into a wiki page.
<div id='myDiv'> <script type="text/javascript" nonce="<%=scriptNonce%>"> var webPart = new LABKEY.WebPart({partName: 'Wiki', renderTo: 'myDiv', partConfig: {name: 'home'} }); webPart.render(); </script>
Retrieve the Rows in a Table
This script retrieves all the rows in a user-created list named "People." Learn more about the parameters used here:
The success and failure callbacks defined in this example illustrate how you might manage the fact that JavaScript requests to LabKey server use AJAX and are asynchronous. You don't get results immediately upon calling a function, but instead at some point in the future, and at that point the success or failure callbacks are run. If you would like to ensure a certain behavior waits for completion, you could place it inside the success callback function as in this example:
var someValue = 'Test value'; LABKEY.Query.selectRows({ schemaName: 'lists', queryName: 'People', columns: ['Name', 'Age'], success: function (data) { alert("Success! " + data.rowCount + " rows returned and value is " + someValue); }, failure: onFailure });
Retrieve a Subset of Rows
Using the parameters to selectRows, you can be even more selective about which parts of the data you retrieve.
Include filters (filterArray)
Change the container scope (containerFilter)
Sort the returned data (sort)
Limit the rows returned (maxRows)
Start the rows you retrieve at some point other than the start (offset)
For example, the following will retrieve 100 rows of the sample type "SampleType1" across the entire project, but starting with the 1000th row. This can be useful for batching retrieval of large tables or isolating problems in scripts that may be related to unexpected data formats that don't appear at the beginning of the table. Note that this is more reliable when you use a 'sort' parameter as well.
var formData = new FormData(); var issues = []; issues.push({ assignedTo : 1016, comment : 'I am not able to repro this bug.', notifyList : 'developerx@labkey.com', //issueDefId: 20, issueDefName: 'mybugs', issueId : 25, action : 'update' }); formData.append('issues', JSON.stringify(issues));
In the above examples, we add a single user to the notifyList by email address. To add multiple users simultaneously, use their userIDs in a semi-colon delimited string:
notifyList : '1016;1005;1017',
Submit an Attached File via the Issues API
<script> function doInsert() { var file = document.getElementById('attachment-file').files[0]; var file2 = document.getElementById('attachment-file2').files[0];
var issues = [{ action : 'insert', issueDefName : 'issues', priority : 3, title : 'test issue 1', comment : 'this is a new issue', attachment : 'attachment.file', type : 'Todo', assignedTo : 1011 },{ action : 'insert', issueDefName : 'issues', priority : 2, title : 'test issue 2', attachment : 'attachment.file2', assignedTo : 1011 }];
Use the LABKEY.Domain.create() API to create a list with the index (or indices) needed. Only a unique field may be used as an index. Shown here, a unique index on the "tubeCode" column that is not the primary key.
LABKEY.Domain.create({ kind: "VarList", //creates a list with the primary key being a text field domainDesign: { name: "storageList", //list name fields: [{ name: "uniqueLocation", rangeURI: "string" },{ name: "boxCode", rangeURI: "string" },{ name: "column", rangeURI: "string" },{ name: "row", rangeURI: "string" },{ name: "tubeCode", // this is the other column we want to index; values are unique rangeURI: "string" }], indices: [{ columnNames: [ "tubeCode" ], //put your column with uniqueness constraint here unique: true }], }, options: { keyName: "uniqueLocation" //put your primary key name here }
});
Rename a Folder
Use the RenameContainer.api to rename a folder. Provide the name, title, and whether to include an alias for the original folder name.
A JavaScript report links a specific data grid with code that runs in the user's browser. The code can access the underlying data, transform it as desired, and render a custom visualization or representation of that data (for example, a chart, grid, summary statistics, etc.) to the HTML page. Once the new JavaScript report has been added, it is accessible from the (Reports) menu on the grid.
Note the "starter code" provided on the Source tab. This starter code simply retrieves the data grid and displays the number of rows in the grid. The starter code also shows the basic requirements of a JavaScript report. Whatever JavaScript code you provide must define a render() function that receives two parameters: a query configuration object and an HTML div element. When a user views the report, LabKey Server calls this render() function to display the results to the page using the provided div.
Modify the starter code, especially the onSuccess(results) function, to render the grid as desired. See an example below.
If you want other users to see this report, place a checkmark next to Make this report available to all users.
Elect whether you want the report to be available in child folders on data grids where the schema and table are the same as this data grid.
Click Save, provide a name for the report, and click OK.
Confirm that the JavaScript report has been added to the grid's Reports menu.
GetData API
There are two ways to retrieve the actual data you wish to see, which you control using the JavaScript Options section of the source editor, circled in red at the bottom of the following screenshot.
If Use GetData APIis selected (the default setting), you can pass the data through one or more transforms before retrieving it. When selected, you pass the query config to LABKEY.Query.GetData.getRawData().
Before the data is retrieved, the query config can be modified as needed. For example, you can specify filters, columns, sorts, maximum number of rows to return, etc. The example below specifies that only the first 25 rows of results should be returned:
queryConfig.maxRows = 25;
Your code should also add parameters to the query configuration to specify functions to call when selectRows succeeds or fails. For example:
function onSuccess(results) { . . .Render results as HTML to div. . . }
function onError(errorInfo) { jsDiv.innerHTML = errorInfo.exception; }
JavaScript Scope
Your JavaScript code is wrapped in an anonymous function, which provides unique scoping for the functions and variables you define; your identifiers will not conflict with identifiers in other JavaScript reports rendered on the same page.
Example
This example can be attached to any dataset or list. To run this sample, select a dataset or list to run it against, create a JavaScript report (see above), pasting this sample code into the Source tab.
var jsDiv;
// When the page is viewed, LabKey calls the render() function, passing a query config // and a div element. This sample code calls selectRows() to retrieve the data from the server, // and displays the data inserting line breaks for each new row. // Note that the query config specifies the appropriate query success and failure functions // and limits the number of rows returned to 4. function render(queryConfig, div) { jsDiv = div; queryConfig.success = onSuccess; queryConfig.error = onError; // Only return the first 4 rows queryConfig.maxRows = 4; LABKEY.Query.GetData.getRawData(queryConfig); //LABKEY.Query.selectRows(queryConfig); }
function onSuccess(results) { var data = "";
// Display the data with white space after each column value and line breaks after each row. for (var idxRow = 0; idxRow < results.rows.length; idxRow++) { var row = results.rows[idxRow];
for (var col in row) { if (row[col] && row[col].value) { data = data + row[col].value + " "; } }
data = data + "<br/>"; }
// Render the HTML to the div. jsDiv.innerHTML = data; }
function onError(errorInfo) { jsDiv.innerHTML = errorInfo.exception; }
LabKey Server provides a rich API for building client applications that retrieve and interact with data from the database. To get started building a client application, LabKey Server can generate a script that retrieves a grid of data from the database. Adapt and extend the script's capabilities to meet your needs. Developers can generate a script snippet for any data grid. The following script languages are supported:
Java
JavaScript
Perl
Python
R
SAS
Stable URL
You can also generate a Stable URL from this export menu which can be used to reload the query, preserving any filters, sorts, or custom sets of columns.
Export/Generate Scripts
To generate a script for a given dataset:
Navigate to the grid view of interest and click (Export).
Select the Script tab and select an available language: Java, JavaScript, Perl, Python, R, or SAS.
Click Create Script to generate a script.
For example, the Physical Exam dataset in the LabKey Demo Study can be retrieved using this snippet of JavaScript:
function onSuccess(results) { var data = ''; var length = Math.min(10, results.rows.length);
// Display first 10 rows in a popup dialog for (var idxRow = 0; idxRow < length; idxRow++) { var row = results.rows[idxRow];
for (var col in row) { data = data + row[col].value + ' '; }
data = data + 'n'; }
alert(data); }
function onError(errorInfo) { alert(errorInfo.exception); }
</script>
Filters. Filters that have been applied to the grid view are included in the script. Note that some module actions apply special filters to the data (e.g., an assay may filter based on a "run" parameter in the URL); these filters are not included in the exported script. Always test the generated script to verify it will retrieve the data you expect, and modify the filter parameters as appropriate.Column List. The script explicitly includes a column list so the column names are obvious and easily usable in the code.Foreign Tables. The name for a lookup column will be the name of the column in the base table, which will return the raw foreign key value. If you want a column from the foreign table, you need to include that explicitly in your view before generating the script, or add "/<ft-column-name>" to the field key.
Use Exported Scripts
JavaScript Examples.
You can paste a script into a <script> block on the source tab of an HTML wiki.
For a better development experience, you can create a custom module. HTML pages in that module can use the script to create custom interfaces.
Use the script within an external R environment to retrieve data from LabKey Server. Paste the script into your R console. Learn more in this topic: Rlabkey Package.
This topic shows you how to create an application providing master-detail page views of some study data. Adapt elements from this sample to create your own similar data dashboard utilities. You must have a developer role to complete these steps, generally on your own local development server.To get started, create a new study folder and import this folder archive:
This archive contains a sample study with extra wikis containing our examples; find them on the Overview tab.
Participant Details View
On the Overview tab of the imported study, click Participant Details View. This wiki displays an Ext4 combo box, into which all the participant IDs from the demographics dataset have been loaded.Use the dropdown to Select a Participant Id. Upon selection, the wiki loads the demographics data as well as several LABKEY.QueryWebParts for various datasets filtered to that selected participant ID.Review the source for this wiki to see how this is accomplished. You can edit the wiki directly if you uploaded the example archive, or download the source here:
The "Participant Grid View" is a Master-Detail page using Ext4 grid panels (LABKEY.Ext4.GridPanel). It loads the demographics dataset and awaits click of a row to load details. Note that if you click the actual participant ID, it is already a link to another participant view in the folder, so click elsewhere in the row to activate this master-detail view.After clicking any non-link portion of the row, you will see a set of other datasets filtered for that participant ID as Ext4 grids.Review the source for this wiki to see how this is accomplished. You can edit the wiki directly if you uploaded the example archive, or download the source here:
The button bar appears by default above data grids and contains icon and text buttons providing various features. You'll find a list of the standard buttons and their functions here.The standard button bars for any query or table can be customized through XML or the JavaScript client API. You can add, replace, delete buttons. Buttons can be linked words, icons, or drop-down menus. You can also control the visibility of custom buttons based on a user's security permissions. Custom button bars can leverage the functionality supplied by default buttons.This topic covers:
This example was used to create a "Custom Dropdown" menu item on the Medical History dataset in the demo study. Two examples of actions that can be performed by custom buttons are included:
Execute an action using the onClick handler ("Say Hello").
A note about positioning: As shown in the example, you can add new buttons while retaining the standard buttons. By default, any new buttons you define are placed after the standard buttons. To position your new buttons along the bar, use one of these buttonBarItem parameters:
insertBefore="Name-of-existing-button"
insertAfter="Name-of-existing-button"
insertPosition="#" OR "beginning" OR "end" : The button positions are numbered left to right starting from zero. insertPosition can place the button in a specific order, or use the string "beginning" or "end".
Require Row Selections
You can set buttons to require that one or more rows are selected before they are activated. Buttons that require selections will be grayed out/inactive when no rows are selected.
requiresSelection="boolean"
requiresSelectionMaxCount="#": the maximum number of rows that can be selected for use with this button. For example, if an action will only work on a single row, set this to "1".
requiresSelectionMinCount="#": the minimum number of rows that need to be selected for the action to work.
Show/Hide Based on Permissions
Using the "hidden" parameter controls whether a button is shown at all. A "requiresPermissions" section setting a "permissionClass" can be used for finer grained control of button visibility. For example, to require that a user have insert permission to see a button, use:
If you want to have a button bar show or hide one or more buttons based on the permission the user has, follow these steps in your XML definition:
First set "includeStandardButtons=false"
Define the buttons you want, setting hidden to be false for all.
Any button that you want to be shown only to users with a specific requiresPermissions -> permissionClass level.
Invoke JavaScript Functions
You can also define a button to invoke a JavaScript function. The function itself must be defined in a .js file included in the resources of a module, typically by using the includeScript element. This excerpt can be used in the XML metadata for a table or query, provided you have "moreActionsHandler" defined.
A custom button can get selected items from the current page of a grid view and perform a query using that info. Note that only the selected options from a single page can be manipulated using onClick handlers for custom buttons. Cross-page selections are not currently recognized.
To see the above example in action, you can install it in your own local build following these steps.If you do not already have a module where you can add this example, create the basic structure of the "HelloWorld" module, and understand the process of building and deploying it following this tutorial first:
Confirm that your module is built, deployed, and then enable it in the folder in which you installed the demo study.
Go to > Folder > Management > Folder Type.
Check the box for "HelloWorld" (or the module you are using).
Click Update Folder.
Still in your demo study folder, select > Go To Module > HelloWorld.
The default 'HelloWorld-begin.view?' is shown. Edit the URL so it reads 'HelloWorld-myDemo.view?' (substituting your module name for HelloWorld if needed) and hit return.
You will see this example, can try the buttons for yourself, and use it as the basis for exploring more options.
To experiment with these custom buttons, simply edit the myDemo.html file, start and stop your server, then refresh your page view to see your changes.
You can insert records into the "Client API Actions" audit log table via the standard LabKey Query APIs, such as LABKEY.Query.insertRows() in the JavaScript client API. For example, you could insert audit records to log when code modifies data in your custom tables, record client-side errors, etc.Logged-in users can insert into the audit log for any folder to which they have read access. Guests cannot insert to the audit table. Rows can only be inserted, they cannot be deleted or updated. A simple example using the JavaScript API:
"Client API Actions" is the only audit log table that allows direct inserts.For details on the LABKEY.Query API, see the documentation for LABKEY.Query.
The table exp.Files is available for users to manage files included under the @files, @pipeline, and @filesets file roots. (Note that file attachments are not included.) You can add custom properties to the File Repository as described in the topic Files Web Part Administration. These custom properties will be added to the exp.Files table, which you can manage programatically using the LabKey APIs as described in this topic.
The table exp.Files is a filtered table over exp.Data which adds a number of columns that aren't available in exp.Data, namely, RelativeFolder and AbsoluteFilePath.To allow access to the AbsoluteFilePath and DataFileUrl fields in exp.Files, assign the site-level role "See Absolute File Paths". For details see Configure Permissions.
JavaScript API Examples
The following example code snippets demonstrate how to use the APIs to control the file system using the exp.Files table.
// List all file records with default columns (including deleted files). LABKEY.Query.selectRows({ schemaName: 'exp', queryName: 'files', success: function(data) { var rows = data.rows; for (var i = 0; i < rows.length; i++) { var row = rows[i]; console.log(row['Name']); } } });
// List all files containing "LabKey" substring in filename, with specified columns, order by Name. LABKEY.Query.selectRows({ schemaName: 'exp', queryName: 'files', columns: ['Name', 'FileSize', 'FileExists', 'RelativeFolder', 'AbsoluteFilePath', 'DataFileUrl'], // Both 'AbsoluteFilePath' and 'DataFileUrl' requires SeeFilePath permission filterArray: [ LABKEY.Filter.create('Name', 'LabKey', LABKEY.Filter.Types.CONTAINS) ], sort: 'Name', success: function(data) { var rows = data.rows; for (var i = 0; i < rows.length; i++) { var row = rows[i]; console.log(row['Name']); } } });
// Query files with custom property fields. // Custom1 & Custom2 are the custom property fields for files. LABKEY.Query.selectRows({ schemaName: 'exp', queryName: 'files', columns: ['Name', 'FileSize', 'FileExists', 'RelativeFolder', 'Custom1', 'Custom2'], filterArray: [ LABKEY.Filter.create('Custom1', 'Assay_Batch_1') ], sort: 'Custom1', success: function(data) { var rows = data.rows; for (var i = 0; i < rows.length; i++) { var row = rows[i]; console.log(row['Name'] + ", " + row['Custom1']); } } });
// Update custom property value for file records. LABKEY.Query.updateRows({ schemaName: 'exp', queryName: 'files', rows: [{ RowId: 1, // update by rowId Custom1: 'Assay_Batch_2' },{ DataFileUrl: 'file:/LabKey/Files/run1.xslx', // update by dataFileUrl Custom1: 'Assay_Batch_2' }], success: function (data) { var rows = data.rows; for (var i = 0; i < rows.length; i++) { var row = rows[i]; console.log(row['RowId'] + ", " + row['Custom1']); } }, failure: function(errorInfo, options, responseObj) { if (errorInfo && errorInfo.exception) alert("Failure: " + errorInfo.exception); else alert("Failure: " + responseObj.statusText); } });
// Insert file records with a custom property. LABKEY.Query.insertRows({ schemaName: 'exp', queryName: 'files', rows: [{ AbsoluteFilePath: '/Users/xing/Downloads/labs.txt', Custom1: 'Assay_Batch_3' }], success: function (data) { var rows = data.rows; for (var i = 0; i < rows.length; i++) { var row = rows[i]; console.log(row['RowId'] + ", " + row['Custom1']); } }, failure: function(errorInfo, options, responseObj) { if (errorInfo && errorInfo.exception) alert("Failure: " + errorInfo.exception); else alert("Failure: " + responseObj.statusText); } });
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
Vocabulary properties can be used to define a dictionary of terms for use in annotating experiment and sample data. These "ad-hoc" terms can capture metadata of interest not known at the time of assay or sample type definition and not present in the data file. Example use cases:
Attaching provenance information to assay data, such as "Instrument type" or "Original data source".
Attaching provenance information to sample data such as "Clinic of origin", etc.
Attaching any information that is not already part of the result data or samples.
Vocabulary properties are defined and stored in a Vocabulary Domain. This topic shows how to create a Vocabulary Domain and attach its properties to assay and sample data.
Using saveRuns to create a new assay run with properties:
LABKEY.Experiment.saveRuns({ assayId: 8753, runs: [{ name: 'testing', properties: { 'urn:lsid:labkey.com:Vocabulary.Folder-3234:ProcessParameters#flowRate': 3.2 }, materialOutputs: [{ id: 1092216 },{ // a new sample will be created when the run is saved name: 'S-123', sampleSet: { name: 'Samples' }, properties: { 'urn:lsid:labkey.com:Vocabulary.Folder-3234:ProcessParameters#flowRate': 3.2 } }], dataRows: [] }] });
Another saveRuns example:
LABKEY.Experiment.saveRuns({ protocolName: LABKEY.Experiment.SAMPLE_DERIVATION_PROTOCOL, runs: [{ name: 'two', properties: { // property URI from a Vocabulary 'urn:lsid:labkey.com:VocabularyDomain.Folder-123:MyVocab#SoftwareTool': 'hello' }, materialInputs: [{ name: ‘ParentSampleA’, sampleSet: {name: 'test'}, properties: { // property name from the Sample Type bar: 3,
// property URI from a Vocabulary 'urn:lsid:labkey.com:VocabularyDomain.Folder-123:MyVocab#SoftwareTool': 'hello', } }], }] });
SQL Queries for Vocabulary Properties
LabKey SQL Example:
SELECT Samples.Name, Samples."urn:lsid:labkey.com:Vocabulary.Folder-3234:MyVocab2#pH", Samples."urn:lsid:labkey.com:Vocabulary.Folder-3234:MyVocab2#instrument", Properties, FROM Samples
Surface in Custom Grid Views
You can surface vocabulary properties in custom grid views as follows:
Go to (Grid Views) > Customize Grid.
Select Show Hidden Fields.
Select Properties to show a nested grid of all properties, or select individual fields to see individual properties. See the screenshot below for examples. Note that rendering the nested Properties column may have performance impacts, as it is backed by a broad query.
Reference Documents
LABKEY.Domain.create - Create a new domain of type "Vocabulary" to store ad-hoc properties.
This topic explains how to declare dependencies on script files, libraries, and other resources. For example, when rendering visualizations, the user should explicitly declare the dependency on LABKEY.vis to load the necessary libraries.
To declare dependencies for all the pages in a module, do the following:First, create a config file named "module.xml" in the resources directory:
myModule/resources/module.xml
Then, add <clientDependencies> and <dependency> tags that point to the required resources. These resources will be loaded whenever a page from your module is called. The path attribute is relative to your /web dir or is an absolute http or https URL. See below for referencing libraries, like Ext4, with the path attribute.
Production-mode builds will created a minified JavaScript file that combines the individual JavaScript files in the client library. Servers running in production mode will serve up the minified version of the .js file to reduce page load times.If you wish to debug browser behavior against the original version of the JavaScript files, you can add the "debugScripts=true" URL parameter to the current page's URL. This will make the server not used the minified version of the resources.
Declare File-Scoped Dependencies
For each HTML file in a file-based module, you can create an XML file with associated metadata. This file can be used to define many attributes, including the set of script dependencies. The XML file allows you to provide an ordered list of script dependencies. These dependencies can include:
JS files
CSS files
libraries
To declare dependencies for HTML views provided by a module, just create a file with the extension '.view.xml' with the same name as your view HTML file. For example, if your view is called 'Overview.html', then you would create a file called 'Overview.view.xml'. An example folder structure of the module might be:
Within the <dependencies> tag, you can list any number of scripts to be loaded. These should be the path to the file, as you might have used previously in LABKEY.requiresScript() or LABKEY.requiresCss(). The example above includes a JS file and a CSS file. These scripts will be loaded in the order listed in this file, so be aware of this if one script depends on another.
In addition to scripts, libraries can be loaded. A library is a collection of scripts. In the example above, the Ext4 library is listed as a dependency. Supported libraries include:
Ext3: Will load the Ext3 library and dependencies. Comparable to LABKEY.requiresExt3()
Ext4: Will load the Ext4 library and dependencies. Comparable to LABKEY.requiresExt4Sandbox()
clientapi: Will load the LABKEY Client API. Comparable to LABKEY.requiresClientAPI()
Declaring dependencies in a .view.xml file is the preferred method of declaring script dependencies where possible. The advantage of declaring dependencies in this manner is that the server will automatically write <script> tags to load these scripts when the HTML view is rendered. This can reduce timing problems that can occur from a dependency not loading completely before your script is processed.An alternative method described below is intended for legacy code and special circumstances where the .view.xml method is unavailable.
Using LABKEY.requiresScript()
From javascript on an HTML view or wiki page, you can load scripts using LABKEY.requiresScript() or LABKEY.requiresCSS(). Each of these helpers accepts the path to your script or CSS resource. In addition to the helpers to load single scripts, LabKey provides several helpers to load entire libraries:
<script type="text/javascript" nonce="<%=scriptNonce%>"> // Require that ExtJS 4 be loaded LABKEY.requiresExt4Sandbox(function() {
// List any JavaScript files here var javaScriptFiles = ["/myModule/Utils.js"];
LABKEY.requiresCss("/myModule/stylesheet.css"); LABKEY.requiresScript(javaScriptFiles, function() { // Called back when all the scripts are loaded onto the page alert("Ready to go!"); }); });
// This is equivalent to what is above LABKEY.requiresScript(["Ext4", "myModule/stylesheet.css", "myModule/Utils.js"], function() { // Called back when all the scripts are loaded onto the page alert("Ready to go!"); }); </script>
Loading Visualization Libraries
To properly render charts and other visualizations, explicitly declare the LABKEY.vis dependency. Using the example of a script with "timeChartHelper.js", the declaration would look like:
// Load the script dependencies for charts. LABKEY.requiresScript('/vis/timeChart/timeChartHelper.js', function(){LABKEY.vis.TimeChartHelper.loadVisDependencies(dependencyCallback);});
Create Custom Client Libraries
If you find that many of your views and reports depend on the same set of javascript or css files, it may be appropriate to create a library of those files so they can be referred to as a group. To create a custom library named "mymodule/mylib", create a new file "mylib.lib.xml" in the web/mymodule directory in your module's resources directory. Just like dependencies listed in views, the library can refer to web resources and other libraries:
Note that external dependencies (i.e. https://.../someScript.js) can only be declared as a dependency of the library, and not as a defining script.
Troubleshooting: Dependencies on Ext3
Past implementations of LabKey Server relied heavily on Ext3, and therefore loaded the ExtJS v3 client API on each page by default. This resulted in the ability to define views, pages, and scripts without explicitly declaring client dependencies. Beginning with version LabKey Server v16.2, DataRegion.js is no longer dependent on ext3, so it is no longer loaded and these views may break at run time.Symptoms: Either a view will fail to operate properly, or a test or script will fail with a JavaScript alert about an undefined function (e.g. "LABKEY.ext.someFn").Workaround: Isolate and temporarily work around this issue by forcing the inclusion of ext3 on every page. Note that this override is global and not an ideal long term solution.
Open > Site > Admin Console.
Under Configuration, click Site Settings.
Check one or both boxes to "Require ExtJS v3… be loaded on each page."
Solutions:Correct views and other objects to explicitly declare their dependencies on client-side resources as described above, or use one of the following overrides:
Override getClientDependencies()
For views that extend HttpView, you can override getClientDependencies() as shown in this example from QueryView.java:
@NotNull @Override public LinkedHashSet<ClientDependency> getClientDependencies() { LinkedHashSet<ClientDependency> resources = new LinkedHashSet<>(); if (!DataRegion.useExperimentalDataRegion()) resources.add(ClientDependency.fromPath("clientapi/ext3")); resources.addAll(super.getClientDependencies()); . . .
Override in .jsp views
Note the <%! syntax when declaring an override as shown in this example from core/project/projects.jsp.
The LabKey JavaScript API provides several extensions to the Ext JavaScript Library. The LABKEY.ext.EditorGridPanel.html is one example.If you use LabKey APIs that extend the Ext API, your code either needs to be open source, or you need to purchase commercial licenses for Ext.For further details, please see the Ext JavaScript licensing page.
Loading ExtJS on Each Page
To load ExtJS on each page of your server:
Go to > Site > Admin Console.
Under Configuration, click Site Settings.
Scroll down to Customize LabKey system properties.
Two checkboxes, for two different libraries, are available:
Require ExtJS v3.4.1 be loaded on each page
Require ExtJS v3.x based Client API be loaded on each page
Note that it is your responsibility to obtain an ExtJS license, if your project does not meet the open source criteria set out by ExtJS. See above for details.
LabKey's JavaScript API reference files are generated automatically when you build LabKey Server. These files can be found in the ROOT\build\client-api\javascript\docs directory, where ROOT is the directory where you have placed the files for your LabKey Server installation.Generating API docs separately can come in handy when you wish to customize the JSDoc compilation settings or alter the JSDoc template. This page helps you generate API reference documentation from annotated javascript files. LabKey uses the open-source JsDoc Toolkit to produce reference materials.
Use the Gradle Build Target
From the ROOT\server directory, use the following to generate the JavaScript API docs:
gradlew jsdoc
You will find the results in the ROOT\build\clientapi_docs folder. Click on the "index.html" file to see your new API reference site.If you need to alter the output template, you can find the JsDoc Toolkit templates in the ROOT\tools\jsdoc-toolkit\templates folder.
Use an Alternative Build Method
You can also build the documents directly from within the jsdoc-toolkit folder.First, place your annotated .js files in a folder called "clientapi" in the jsdoc-toolkit folder (<JSTOOLKIT> in the code snippet below). Then use a command line similar to the following to generate the docs:
You will find the resulting API doc files a folder called "out" in your jsdocs-toolkit folder. Click on the "index.html" file inside the jsdocs folder inside "out" to see your new API reference site.
Further Info on JsDocs and Annotating Javascript with Tags
This topic lists recommendations for writing JSDoc annotations.1. Follow LabKey's JavaScript API naming guidelines.2. When documenting objects that are not explicitly included in the code (e.g., objects passed via successCallbacks), avoid creating extra new classes.
Ideally, document the object inline as HTML list in the method or field that uses it.
If you do need to create an arbitrary class to describe an object, use the @name tag. You'll probably need to create a new class to describe the object IF:
Many classes use the object, so it's confusing to doc the object inline in only one class.
The object is used as the type of many other variables.
The object has (or will have) both methods and fields, so it would be hard to distinguish them in a simple HTML list.
3. Caution: Watch for a bug if you use metatags to write annotations once and use them across a group of enclosed APIs. If you doc multiple, similar, objects that have field names in common, you may have to fully specify the name of the field-in-common. If this bug is problematic, fields that have the same names across APIs will not show links.
4. When adding a method, event or field, please remember to check whether it is correctly marked static.
There are two ways to get a method to be marked static, depending on how the annotations are written:
Leave both "prototype" and "#" off of the end of the @scope statement (now called @lends) for a @namespace
Leave both "prototype" and "#" off of the end of the @method statement
Note: If you have a mix of static and nonstatic fields/methods, you may need to use "prototype" or "#" on the end of a @fieldOf or @memeberOf statement to identify nonstatic fields/methods.
As of 9.3, statics should all be marked correctly.
5. Check out the formatting of @examples you’ve added – it’s easy for examples to overflow the width of their boxes, so you may need to break up lines.
6. Remember to take advantage of LabKey-defined objects when defining types instead of just describing the type as an {Object}. This provides cross-linking.
7. Use @link often to cross-reference classes. For details on how to correctly reference instance vs. static objects, see NamePaths.
8. Cross-link to the main doc tree on labkey.org whenever possible.
9. Deprecate classes using a red font. See GridView for an example. Note that a bug in the toolkit means that you may need to hard-code the font color for the class that’s listed next in the class index.
This topic covers recommended patterns for naming methods, fields, properties and classes in our JavaScript APIs. Capitalization guidelines have been chosen for consistency with our existing JavaScript APIs.
The web directory is shared across all modules so it is a best practice to place your module's resources under a unique directory name within the web directory. It is usually sufficient to use your module's name to scope your resources. For example,
These follow Ext naming conventions.Listener method names
Document failure as the name of a method that listens for errors.
Also support: failureCallback and errorCallback but not "errorListener"
Document success as the name of a method that listens for success.
Also support: successCallback
Failure listener arguments
Use error as the first parameter (not "errorInfo" or "exceptionObj"). This should be a JavaScript Error object caught by the calling code.
This object should have a message property (not "exception").
Use response as the second parameter (not "request" or "responseObj"). This is the XMLHttpRequest object that generated the request. Make sure to say "XMLHttpRequest" in explaining this parameter, not "XMLHttpResponse," which does not exist.
Use Consistent Capitalization
General Guidelines
Use UpperCamelCase for the names of classes.
Use lowercase for the names of events.
Use lowerCamelCase for the names of methods, fields and properties. See the special cases for acronyms below.
The client-side library for Java developers is a separate JAR from the LabKey Server codebase. It can be used by any Java program, including another Java web application.
Premium Feature — Available in the Professional and Enterprise Editions of LabKey Server. Learn more or contact LabKey.
The JDBC driver for LabKey Server allows client applications to query against the schemas, tables, and queries that LabKey Server exposes using LabKey SQL. It implements a subset of the full JDBC functionality, supporting read only (SELECT) access to the data. Update, insert, and delete operations are not supported.Containers (projects and folders) are exposed as JDBC catalogs. Schemas within a given container are exposed as JDBC schemas.
Other tools may work with the driver as it is currently implemented, but some tools may require driver functionality that has not yet been implemented.
Acquire the JDBC Driver
The JDBC driver is bundled with the connectors module and distributed in the Professional and Enterprise distributions of LabKey Server, as an add-on. Please contact your Account Manager if you need assistance in locating the driver.To download the driver:
Log in to your Premium Edition LabKey Server with administrator credentials.
Select > Site > Admin Console.
Click External Analytics Connections under Premium Features.
Check the box to Allow Connections and Save.
Select External Tool Access under the user menu.
You'll find buttons for downloading both the LabKey and Postgres JDBC Drivers.
Click Download LabKey JDBC Driver.
Note that this driver jar also contains the LabKey Java client api and all of its dependencies.
Driver Usage
Driver class: org.labkey.jdbc.LabKeyDriver
Database URL: The base URL of the web server, including any context path, prefixed with "jdbc:labkey:". This value is shown in the panel above the button you clicked to download. Examples include "jdbc:labkey:http://localhost:8080/" and "jdbc:labkey:https://www.labkey.org/". You may include a folder path after a # to set the default target, without the need to explicitly set a catalog through JDBC. For example, "jdbc:labkey:http://localhost:8080/#/MyProject/MyFolder"
Username: Associated with an account on the web server
Password: Associated with an account on the web server
Properties
The driver supports the following properties, which can be set either in Java code by setting in the Properties handed to to DriverManager.getConnection(), or by setting on the Connection that is returned by calling setClientInfo().
rootIsCatalog - Setting rootIsCatalog true will force the root container on the server to only be exposed as a catalog in calls to getTables(). Otherwise the schemas/tables will also be exposed at the top level of the tree. Note that folders and projects in LabKey Server are exposed as individual catalogs (databases) through the jdbc driver. Ordinarily we might expose the schemas for the LabKey Server root container at both the top level of a tree, and in a catalog with name "/". This can be problematic if the user connecting doesn’t have permissions to the root container (i.e., is not an admin); attempting to enumerate the top level schemas results in a 403 (Forbidden) response. Setting the "rootIsCatalog" flag true will cause the driver to skip enumerating the top level schemas, and only expose root as a catalog.
timeout - In DbVisualizer, set the Timeout in the Properties tab on the connection configuration. The default timeout is 60 seconds for any JDBC command. You may set it to 0 to disable the timeout, or the specific timeout you'd like, in milliseconds.
containerFilter - Specify a container (folder) filter for queries to control what folders and subfolders of data will be queried. Possible values are:
Current (Default)
CurrentAndSubfolders
CurrentPlusProject
CurrentAndParents
CurrentPlusProjectAndShared
AllFolders
ignoreNetrc - when set to "true", prevents the driver from using credentials from the user's .netrc file when present. Added in 1.1.0 of the driver.
Integration with AWS Glue: AWS Glue integration is also supported using the Postgres JDBC driver that is preconfigured within Glue to connect to LabKey as a data source.
Premium Feature — Available in the Professional and Enterprise Editions of LabKey Server. Also available as an Add-on to the Starter Edition. Learn more or contact LabKey.
LabKey's JDBC driver allows DBVisualizer to query the server and explore the schemas.To configure DBVisualizer to use the LabKey JDBC driver, do the following:
Creation and updates of security groups and role assignments may be scripted and performed automatically using the LabKey Security API. New user IDs are automatically created as needed.
Ensure Group and Update, Replace, or Delete Members
Group members can be specified in one of these ways:
email - specifies a user; if the user does not already exist in the system, it will be created and will be populated with any of the additional data provided
userId - specifies a user already in the system. If the user does not already exist, this will result in an error message for that member. If both email and userId are provided, this will also result in an error.
groupId - specifies a group member. If the group does not already exist, this will result in an error message for that member.
public static class GroupForm { private Integer _groupId; // Nullable; used first as identifier for group; private String _groupName; // Nullable; required for creating a group private List<GroupMember> _members; // can be used to provide more data than just email address; can be empty; // can include groups, but group creation is not recursive private Boolean _createGroup = false; // if true, the group should be created if it doesn't exist; //otherwise the operation will fail if the group does not exist private MemberEditOperation _editOperation; // indicates the action to be performed with the given users in this group
}
public enum MemberEditOperation { add, // add the given members; do not fail if already exist replace, // replace the current members with the new list (same as delete all then add) delete, // delete the given members; does not fail if member does not exist in group; //does not delete group if it becomes empty };
A response from a successful operation will include the groupId, groupName, a list of users that were added to the system, lists of members added or removed from the group, as well as a list of members if any, that had errors:
LabKey's Perl API allows you to query, insert and update data on a LabKey Server from Perl. The API provides functionality similar to the following LabKey JavaScript APIs:
The netrc file provides credentials for the API to use to authenticate to the server, required to read or modify tables in secure folders.
Python API
LabKey's Python APIs allow you to query, insert and update data on a LabKey Server from Python.A good way to get started with writing a Python script to access data is to view that data in LabKey, then select (Export) > Script > Python and Export Script. This starter script will show you the correct call to APIWrapper to get you started with a basic select.Detailed documentation is available on GitHub:
The LabKey client library for R makes it easy for R users to load live data from a LabKey Server into the R environment for analysis, provided users have permissions to read the data. It also enables R users to insert, update, and delete records stored on a LabKey Server, provided they have appropriate permissions to do so. The Rlabkey APIs use HTTP requests to communicate with a LabKey Server.
All requests to the LabKey Server using the R API are performed under the user's account profile, with all proper security enforced on the server. User credentials can be provided via a netrc file rather than directly in the running R program, making it possible to write reports and scripts that can be shared among many users without compromising security.Learn more about authentication options in these topics:
If you wish, you can also include authentication credentials (either an apikey or full email and password) directly in your R script by using labkey.setDefaults. Learn more in the Rlabkey documentation on CRANTroubleshoot common issues using the information in this topic: Troubleshoot Rlabkey
Install the Rlabkey package once using the following command in the R console. (You may want to change the value of repos depending on your geographical location.)
Necessary if you wish to modify a password-protected LabKey Server database through the Rlabkey macros.
Note that Rlabkey handles sessionid and authentication internally. Rlabkey passes the sessionid as an HTTP header for all API calls coming from that R session. LabKey Server treats this just as it would a valid JSESSIONID parameter or cookie coming from a browser.
Scenarios
The Rlabkey package supports the transfer of data between a LabKey Server and an R session.
Retrieve data from LabKey into a data frame in R by specifying the query schema information (labkey.selectRows and getRows) or by using SQL commands (labkey.executeSql).
Update existing data from an R session (labkey.updateRows).
Insert new data either row by row (labkey.insertRows) or in bulk (labkey.importRows) via the TSV import API.
Delete data from the LabKey database (labkey.deleteRows).
Use Interactive R to discover available data via schema objects (labkey.getSchema).
For example, you might use an external instance of R to do the following:
Connect to LabKey Server.
Use queries to show which schemas and datasets/lists/queries are available within a specific project or sub-folder.
Create colSelect and colFilter parameters for the labkey.selectRows command on the selected schema and query.
Retrieve a data frame of the data specified by the current url, folder, schema, and query context.
Perform transformations on this data frame locally in your instance of R.
Save the revised data frame back into the desired target on LabKey Server.
Within the LabKey interface, the Rlabkey macros are particularly useful for accessing and manipulating datasets across folders and projects.
Run R Scripts on a Schedule
If you want to configure an R script to run locally on a schedule, consider using an R scheduler package such as:
cronR (on Mac)
taskschedulerR (on Windows)
Note that such schedulers will not work for a script defined and running within LabKey Server itself. As an alternative, consider options that will run your script during data import instead:
Premium Feature AvailableSubscribers to premium editions of LabKey Server have the additional option of using ETLs to automate and schedule transformations of data. Learn more in these topics:
If you have an R report that will run in the background and does not need to directly transform data during the ETL, consider adding an R report run task as described here to "kick off" that report on the schedule of your choosing:
This topic provides basic diagnostic tests and solutions to common connection errors related to configuring the Rlabkey package to work with LabKey Server.
The following will gather basic information about the R configuration on the server. Run the following in an R view. To create an R view: from any data grid, select Report > Create R Report.
library(Rlabkey) cat("Output of SessionInfo \n") sessionInfo() cat("\n\n\nOutput of Library Search path \n") .libPaths()
This will output important information such as the version of R being run, the version of each R library, the Operating System of the server, and the location of where the R libraries are being read from.Check that you are running a modern version of R, and using the latest version of Rlabkey and RCurl. If anything is old, we recommend that you update the packages.
Test HTTPS Connection
The following confirms that R can make a HTTPS connection to known good server. Run the following in an R View:
library(Rlabkey) library(RCurl) cat("\n\nAttempt a connection to Google. If it works, print first 200 characters of website. \n") x = getURLContent("https://www.google.com") substring(x,0,200)
If this command fails, then the problem is with the configuration of R on your server. If the server is running Windows, the problem is most likely that their are no CA Certs defined. You will need to fix the configuration of R to ensure a CA Certificate is defined. Use the RLABKEY_CAINFO_FILE environment variable. See http://finzi.psych.upenn.edu/library/Rlabkey/html/labkey.setCurlOptions.html
Diagnose RCurl or Rlabkey
Next check if the problem is coming from the RCurl library or the Rlabkey library. Run the following in an R View, replacing "DOMAIN.org" with your server:
library(Rlabkey) cat("\n\n\nAttempt a connection to DOMAIN.org using only RCurl. If it works, print first 200 characters of website. \n") y = getURLContent("https://DOMAIN.org:8443") substring(y,0,200)
If this command fails, it means there is a problem with the SSL Certificate installed on the server.
Certificate Test
The 4th test is to have R ignore any problems with certificate name mis-matches and certificate chain integrity (that is, using a self-signed certificate or the certificate is signed by a CA that the R program does not trust. In an R view, add the following line after library(Rlabkey)
If this command fails, then there is a problem with the certificate. A great way to see the information on the certificate is to run the following from Linux or OSX:
This will show all certificates in the cert chain and whether they are trusted. If you see verify return:0 near the top of the output then the certificate is good.
Common Issues
Syntax Change from . to _
The syntax for arguments to setCurlOptions has changed. If you see an error like this:
Error in labkey.setCurlOptions(ssl.verifypeer = FALSE, ssl.verifyhost = FALSE) : The legacy config : ssl.verifyhost is no longer supported please update to use : ssl_verifyhost Execution halted
Use the arguments set_verifypeer and set_verifyhost instead.
TLSv1 Protocol Replaces SSLv3
By default, Rlabkey will connect to LabKey Server using the TLSv1 protocol. If your attempt to connect fails, you might see an error message similar to one of these:
Error in function (type, msg, asError = TRUE) : error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
Error in function (type, msg, asError = TRUE) : error:1411809D:SSL routines:SSL_CHECK_SERVERHELLO_TLSEXT:tls invalid ecpointformat list
First confirm that you are using the latest versions of Rlabkey and RCurl, both available on CRAN.If you still encounter this issue, you can add the following to your R scripts or R session. This command tells R to use the TLSv1+ protocol (instead of SSLv3) for all connections:
labkey.setCurlOptions(sslversion=1)
Tomcat GET Header Limit
By default, Tomcat sets a size limit of 4096 bytes (4 KB) for the GET header. If your API calls hit this limit, you can increase the default header size for GETs. Learn more in this troubleshooting section:
Rlabkey uses the package RCurl to connect to the LabKey Server. On Windows, older versions of the RCurl package are not configured for SSL by default. In order to connect, you may need to perform the following steps:1. Create or download a "ca-bundle" file.We recommend using ca-bundle file that is published by Mozilla. See http://curl.haxx.se/docs/caextract.html. You have two options:
2. Copy the ca-bundle.crt file to a location on your hard-drive.If you will be the only person using the Rlabkey package on your computer, we recommend that you
create a directory named `labkey` in your home directory
copy the ca-bundle.crt into the `labkey` directory
If you are installing this file on a server where multiple users will use may use the Rlabkey package, we recommend that you
create a directory named `c:\labkey`
copy the ca-bundle.crt into the `c:\labkey` directory
3. Create a new Environment variable named `RLABKEY_CAINFO_FILE`On Windows 7, Windows Server 2008 and earlier
Select Computer from the Start menu.
Choose System Properties from the context menu.
Click Advanced system settings > Advanced tab.
Click on Environment Variables.
Under System Variables click on the new button.
For Variable Name: enter RLABKEY_CAINFO_FILE
For Variable Value: enter the path of the ca-bundle.crt you created above.
Hit the Ok buttons to close all the windows.
On Windows 8, Windows 2012 and above
Drag the Mouse pointer to the Right bottom corner of the screen.
Click on the Search icon and type: Control Panel.
Click on -> Control Panel -> System and Security.
Click on System -> Advanced system settings > Advanced tab.
In the System Properties Window, click on Environment Variables.
Under System Variables click on the new button.
For Variable Name: enter RLABKEY_CAINFO_FILE
For Variable Value: enter the path of the ca-bundle.crt you created above.
Hit the Ok buttons to close all the windows.
Now you can start R and begin working.
Self-Signed Certificate Authentication
If you are using a self-signed certificate, and connecting via HTTPS on a OSX or Linux machine, you may see the following issues as Rlabkey attempts unsuccessfully to validate that certificate.
Peer Verification
If you see an error message that looks like the following, you can tell Rlabkey to ignore any failures when checking if the server's SSL certificate is authentic.
> rows <- labkey.selectRows(baseUrl="https://SERVERNAME", folderPath="home",schemaName="lists", queryName="myFavoriteList") Error in function (type, msg, asError = TRUE) : SSL certificate problem, verify that the CA cert is OK. Details: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
To bypass the peer verification step, add the following to your script:
labkey.setCurlOptions(ssl_verifypeer=FALSE)
Certificate Name Conflict
It is possible to tell Rlabkey to ignore any failures when checking if the server's name used in baseURL matches the one specified in the SSL certificate. An error like the following could occur when the name on the certificate is different than the SERVERNAME used.
> rows <- labkey.selectRows(baseUrl="https://SERVERNAME", folderPath="home",schemaName="lists", queryName="ElispotPlateReader") Error in function (type, msg, asError = TRUE) : SSL peer certificate or SSH remote key was not OK
To bypass the host verification step, add the following to your script:
labkey.setCurlOptions(ssl_verifyhost=FALSE)
Troubleshoot Authentication
Rlabkey functions can be authenticated using an API key or username and password, either via a netrc file or using labkey.setDefaults() to set both username and password. Update to Rlabkey version 2.3.4, available on CRAN.API keys, or session keys tied to a single browser session, must be enabled on your server by an administrator. When a key is generated and then used in Rlabkey, it will be used to authenticate operations based on the access available to the user who generated the key at the time the key was generated.If an API key is not provided, the labkey.setDefaults() function can be used for basic email/password authentication. Both must be set simultaneously. Note that if both an apikey and a username/password combination are supplied, the apikey will take precendence and the username and password will be ignored.Learn more:
The LabKey Client API Library for SAS makes it easy for SAS users to load live data from a LabKey Server into a native SAS dataset for analyis. It also enables SAS users to insert, update, and delete records stored on a LabKey Server.Any access to read or modify data in SAS datasets is performed under the user's LabKey Server account profile, with all proper security enforced on the server. Users only have access to read and/or modify data in SAS that they are authorized to view or modify within LabKey Server. User credentials are obtained from a separate location than the running SAS program so that SAS programs can be shared without compromising security.The SAS macros use the Java Client Library to send, receive, and process requests to the server. They provide functionality similar to the Rlabkey Package.
The SAS library performs all requests to the LabKey Server under a given user account with all the proper security enforced on the server. User credentials are obtained from a separate location than the running SAS program so that SAS programs may be shared without compromising security.User credentials are read from the netrc file in the user’s home directory, so as to keep those credentials out of SAS programs, which may be shared between users.
Premium Feature — Available with all Premium Editions of LabKey Server. Learn more or contact LabKey.
The LabKey/SAS client library is a set of SAS macros that retrieve data from an instance of LabKey Server as SAS data sets. The SAS macros use the Java Client Library to send, receive, and process requests to the server. This topic helps you set up SAS to work with LabKey.
Configure your SAS Installation to Use the SAS/LabKey Interface
The steps in this section illustrate the process; be sure to substitute your own current version numbers for the examples given.
Install SAS
Get the latest SAS client distribution, as follows:
Go to the topic API Resources, which displays a table of API resources.
In the row of SAS related links, click Distribution, which takes you to the artifact publication site.
Click the directory for the latest release (1.0.1 as of this writing) then download the .zip file, for example labkey-api-sas-1.0.1.zip.
Extract this file to a local directory (these instructions assume "c:\sas"). The directory should contain a number of .jar files (the Java client library and its dependencies) and 12 .sas files (the SAS macros).
Open your default SAS configuration file, sasv9.cfg. You will find it in your SAS installation in a location similar to C:\Program Files\SASHome\x86\SASFoundation\9.3\nls\en)
In the -SET SASAUTOS section, add the path to the SAS macros to the end of the list (e.g., "C:\sas")
Configure your Java Runtime Environment (JRE) based on your SAS version. For current information, check the SAS documentation. You may need to install security updates or make other adjustments as recommended there.
Near the top of sasv9.cfg, add a line like the following, including a path list surrounded by double quotes. The path list must use the correct separator for your operating system. For Windows, use a semicolon ";" and for Mac, us a colon ":"
-set classpath "[full paths to all .jar files]"
Example Java classpath for Windows (use the actual version numbers from the package you unpacked):
On your local version of LabKey Server, create a list called "People" in your home folder, inferring the structure and populating it using the file "demo.xls".
Execute "proc javainfo; run;" in a program editor; this command should display detailed information about the java environment in the log. Verify that java.version matches the JRE you set above.
The SAS/LabKey client library provides a set of SAS macros that retrieve data from an instance of LabKey Server as SAS data sets and allows modifications to LabKey Server data from within SAS. All requests to the LabKey Server are performed under the user's account profile, with all proper security enforced on the server.The SAS macros use the Java Client Library to send, receive and process requests to the server. This page lists the SAS macros, parameters and usage examples.
The %labkeySetDefaults macro sets connection information that can be used for subsequent requests. All parameters set via %labkeySetDefaults can be set once via %labkeySetDefaults, or passed individually to each macro.The %labkeySetDefaults macro allows the SAS user to set the connection information once regardless of the number of calls made. This is convenient for developers, who can write more maintainable code by setting defaults once instead of repeatedly setting these parameters.Subsequent calls to %labkeySetDefaults will change any defaults set with an earlier call to %labkeySetDefaults.%labkeySetDefaults accepts the following parameters:
Name
Type
Required?
Description
baseUrl
string
n
The base URL for the target server. This includes the protocol (http, https) and the port number. It will also include the context path (commonly "/labkey"), unless LabKey Server has been deployed as the root context. Example: "http://localhost:8080/labkey"
folderPath
string
n
The LabKey Server folder path in which to execute the request
schemaName
string
n
The name of the schema to query
queryName
string
n
The name of the query to request
userName
string
n
The user's login name. Note that the NetRC file includes both the userName and password. It is best to use the values stored there rather than passing these values in via a macro because the passwords will show up in the log files, producing a potential security hole. However, for chron jobs or other automated processes, it may be necessary to pass in userName and password via a macro parameter.
password
string
n
The user's password. See userName (above) for further details.
containerFilter
string
n
This parameter modifies how the query treats the folder. The possible settings are listed below. If not specified, "Current" is assumed.
Options for the containerFilter parameter:
Current -- The current container
CurrentAndSubfolders -- The current container and any folders it contains
CurrentPlusProject -- The current container and the project folder containing it
CurrentAndParents -- The current container and all of its parent containers
CurrentPlusProjectAndShared -- The current container, its project folder and all shared folders
AllFolders -- All folders to which the user has permission
The %labkeySelectRows macro allows you to select rows from any given schema and query name, optionally providing sorts, filters and a column list as separate parameters.Parameters passed to an individual macro override the values set with %labkeySetDefaults.Parameters are listed as required when they must be provided either as an argument to %labkeySelectRows or through a previous call to %labkeySetDefaults.This macro accepts the following parameters:
Name
Type
Required?
Description
dsn
string
y
The name of the SAS dataset to create and populate with the results
baseUrl
string
y
The base URL for the target server. This includes the protocol (http, https), the port number, and optionally the context path (commonly "/labkey"). Example: "http://localhost:8080/labkey"
folderPath
string
y
The LabKey Server folder path in which to execute the request
schemaName
string
y
The name of the schema to query
queryName
string
y
The name of the query to request
viewName
string
n
The name of a saved custom grid view of the given schema/query. If not supplied, the default grid will be returned.
filter
string
n
One or more filter specifications created using the %makeFilter macro
columns
string
n
A comma-delimited list of column name to request (if not supplied, the default set of columns are returned)
sort
string
n
A comma-delimited list of column names to sort by. Use a “-“ prefix to sort descending.
maxRows
number
n
If set, this will limit the number of rows returned by the server.
rowOffset
number
n
If set, this will cause the server to skip the first N rows of the results. This, combined with the maxRows parameter, enables developers to load portions of a dataset.
showHidden
1/0
n
By default hidden columns are not included in the dataset, but the SAS user may pass 1 for this parameter to force their inclusion. Hidden columns are useful when the retrieved dataset will be used in a subsequent call to %labkeyUpdate or %labkeyDetele.
userName
string
n
The user's login name. Please see the %labkeySetDefaults section for further details.
password
string
n
The user's password. Please see the %labkeySetDefaults section for further details.
containerFilter
string
n
This parameter modifies how the query treats the folder. The possible settings are listed in the %labkeySetDefaults macro section. If not specified, "Current" is assumed.
Examples:The SAS code to load all rows from a list called "People" can define all parameters in one function call:
Alternatively, default parameter values can be set first with a call to %labkeySetDefaults. This leaves default values in place for all subsequent macro invocations. The code below produces the same output as the code above:
This example demonstrates column list, column sort, row limitation, and row offset:
%labkeySelectRows(dsn=limitRows, columns="First, Last, Age", sort="Last, -First", maxRows=3, rowOffset=1);
Further examples are available in the %labkeyMakeFilter section below.
The %labkeyMakeFilter Macro
The %labkeyMakeFilter macro constructs a simple compare filter for use in the %labkeySelectRows macro. It can take one or more filters, with the parameters listed in triples as the arguments. All operators except "MISSING and "NOT_MISSING" require a "value" parameter.
Name
Type
Required?
Description
column
string
y
The column to filter upon
operator
string
y
The operator for the filter. See below for a list of acceptable operators.
value
any
y
The value for the filter. Not used when the operator is "MISSING" or "NOT_MISSING".
The operator may be one of the following:
EQUAL
NOT_EQUAL
NOT_EQUAL_OR_MISSING
DATE_EQUAL
DATE_NOT_EQUAL
MISSING
NOT_MISSING
GREATER_THAN
GREATER_THAN_OR_EQUAL
LESS_THAN
LESS_THAN_OR_EQUAL
CONTAINS
DOES_NOT_CONTAIN
STARTS_WITH
DOES_NOT_START_WITH
IN
NOT_IN
CONTAINS_ONE_OF
CONTAINS_NONE_OF
Note: For simplicity and consistency with other client libraries, EQUALS_ONE_OF has been renamed IN and EQUALS_NONE_OF has been renamed NOT_IN. You may need to update your code to support these new filter names.Examples:
/* Specify two filters: only males less than a certain height. */ %labkeySelectRows(dsn=shortGuys, filter=%labkeyMakeFilter("Sex", "EQUAL", 1, "Height", "LESS_THAN", 1.2)); proc print label data=shortGuys; run;
/* Demonstrate an IN filter: only people whose age is specified. */ %labkeySelectRows(dsn=lateThirties, filter=%labkeyMakeFilter("Age", "IN", "36;37;38;39")); proc print label data=lateThirties; run;
/* Specify a grid and a not missing filter. */ %labkeySelectRows(dsn=namesByAge, viewName="namesByAge", filter=%labkeyMakeFilter("Age", "NOT_MISSING")); proc print label data=namesByAge; run;
The %labkeyExecuteSql Macro
The %labkeyExecuteSql macro allows SAS users to execute arbitrary LabKey SQL, filling a SAS dataset with the results.Required parameters must be provided either as an argument to %labkeyExecuteSql or via a previous call to %labkeySetDefaults.This macro accepts the following parameters:
Name
Type
Required?
Description
dsn
string
y
The name of the SAS dataset to create and populate with the results
sql
string
y
The LabKey SQL to execute
baseUrl
string
y
The base URL for the target server. This includes the protocol (http, https), the port number, and optionally the context path (commonly "/labkey"). Example: "http://localhost:8080/labkey"
folderPath
string
y
The folder path in which to execute the request
schemaName
string
y
The name of the schema to query
maxRows
number
n
If set, this will limit the number of rows returned by the server.
rowOffset
number
n
If set, this will cause the server to skip the first N rows of the results. This, combined with the maxrows parameter, enables developers to load portions of a dataset.
showHidden
1/0
n
Please see description in %labkeySelectRows.
userName
string
n
The user's login name. Please see the %labkeySetDefaults section for further details.
password
string
n
The user's password. Please see the %labkeySetDefaults section for further details.
containerFilter
string
n
This parameter modifies how the query treats the folder. The possible settings are listed in the %labkeySetDefaults macro section. If not specified, "Current" is assumed.
Example:
/* Set default parameter values to use in subsequent calls. */ %labkeySetDefaults(baseUrl="http://localhost:8080/labkey", folderPath="/home", schemaName="lists", queryName="People");
/* Query using custom SQL… GROUP BY and aggregates in thiscase. */ %labkeyExecuteSql(dsn=groups, sql="SELECT People.Last, COUNT(People.First) AS Number, AVG(People.Height) AS AverageHeight, AVG(People.Age) AS AverageAge FROM People GROUP BY People.Last"); proc print label data=groups; run;
/* Demonstrate UNION between two different data sets. */ %labkeyExecuteSql(dsn=combined, sql="SELECT MorePeople.First, MorePeople.Last FROM MorePeople UNION SELECT People.First, People.Last FROM People ORDER BY 2"); proc print label data=combined; run;
The %labkeyInsertRows, %labkeyUpdateRows and %labkeyDeleteRows Macros
The %labkeyInsertRows, %labkeyUpdateRows and %labkeyDeleteRows macros are all quite similar. They each take a SAS dataset, which may contain the data for one or more rows to insert/update/delete.Required parameters must be provided either as an argument to %labkeyInsert/Update/DeleteRows or via a previous call to %labkeySetDefaults.Parameters:
Name
Type
Required?
Description
dsn
dataset
y
A SAS dataset containing the rows to insert/update/delete
baseUrl
string
y
The base URL for the target server. This includes the protocol (http, https), the port number, and optionally the context path (commonly "/labkey"). Example: "http://localhost:8080/labkey"
folderpath
string
y
The folder path in which to execute the request
schemaName
string
y
The name of the schema
queryName
string
y
The name of the query within the schema
userName
string
n
The user's login name. Please see the %labkeySetDefaults section for further details.
password
string
n
The user's password. Please see the %labkeySetDefaults section for further details.
The key difference between the macros involves which columns are required for each case.
For insert, the input dataset should not include values for the primary key column ('lsid' for study datasets), as this will be automatically generated by the server.For update, the input dataset must include values for the primary key column so that the server knows which row to update. The primary key value for each row is returned by %labkeySelectRows and %labkeyExecuteSql if the 'showHidden' parameter is set to 1.For delete, the input dataset needs to include only the primary key column. It may contain other columns, but they will be ignored by the server.Example: The following code inserts new rows into a study dataset:
/* Set default parameter values to use in subsequent calls. */ %labkeySetDefaults(baseUrl="http://localhost:8080/labkey", folderPath="/home", schemaName="lists", queryName="People");
data children; input First : $25. Last : $25. Appearance : mmddyy10. Age Sex Height ; format Appearance DATE9.; datalines; Pebbles Flintstone 022263 1 2 .5 Bamm-Bamm Rubble 100163 1 1 .6 ;
/* Insert the rows defined in the children data set into the "People" list. */ %labkeyInsertRows(dsn=children);
Quality Control Values
The SAS library accepts special values in datasets as indicators of the quality control status of data. The QC values currently available are:
'Q': Data currently under quality control review
'N': Required field marked by site as 'data not available'
The SAS library will save these as “special missing values” in the data set.
You can select (Export) > Script > SAS above most query views to export a script that selects the columns shown.For example, performing this operation on the LabResults dataset in the example study, produces the following macro:
%labkeySelectRows(dsn=mydata, baseUrl="https://www.labkey.org", folderPath="/home/Demos/HIV Study Tutorial", schemaName="study", queryName="LabResults");
This SAS macro selects the rows shown in this custom grid into a dataset called 'mydata'.
Full SAS Demo
The sas-demo.zip archive attached to this page provides a SAS script and Excel data files. You can use these files to explore the selectRows, executeSql, insert, update, and delete operations of the SAS/LabKey Library.Steps for setting up the demo:
Make sure that you or your admin has set up a .netrc file to provide you with appropriate permissions to insert/update/delete. For further information, see Create a netrc file.
Download and unzip the demo files: sas-demo.zip. The zip folder contains a SAS demo script (demo.sas) and two data files (People.xls and MorePeople.xls). The spreadsheets contain demo data that goes with the script.
Create a new list called “People”, inferring the fields and importing the data by dragging and dropping the file People.xls into the list creation editor. Learn more in this topic: Create Lists.
Create a second list called “MorePeople” using MorePeople.xls for inferring fields and importing data.
Change the two references to baseUrl and folderPath in the demo.sas to match your server and folder.
LabKey client libraries provide flexible authentication mechanisms and automatically handle cookies & sessions, CSRF tokens, marshalling of parameters & payloads, and returning results in native data structures that are easy to manipulate. We strongly recommend using them to interact programmatically with LabKey Server.If absolutely required (e.g., a client library does not exist for your preferred language), you can interact with a LabKey Server through direct HTTP requests, as covered in this topic. Note that this requires significantly more effort than using a LabKey client library.
The HTTP Interface exposes a set of API endpoints that accept parameters & JSON payloads and return JSON results. These may be called from any program capable of making an HTTP request and decoding the JSON format used for responses (e.g., C++, C#, etc.).This document describes the API actions that can be used by HTTP requests, detailing their URLs, inputs and outputs. For information on using the JavaScript helper objects within web pages, see JavaScript API. For an example of using the HTTP Interface from Perl, see Example: Access APIs from Perl.
Calling API Actions from Client Applications and Scripts
The API actions documented below may be used by any client application or script capable of making an HTTP request and handling the response. Consult your programming language’s or operating environment’s documentation for information on how to submit an HTTP request and process the response. Most modern languages include HTTP and JSON libraries or helpers.Several actions accept or return information in the JavaScript Object Notation (JSON) format, which is widely supported. See https://json.org for information on the format, and to obtain libraries/plug-ins for most languages.
Basic Authentication
Most API actions require the user to be authenticated so that the correct permissions can be evaluated. Clients should use basic authentication over HTTPS so that the headers will be encrypted.LabKey Server uses form-based authentication by default for all user-agents (browsers). However, it will also accept http basic authentication headers if presented. This can be useful for command line tools that you might use to automate certain tasks.For instance, to use wget to retrieve a page readable by 'user1' with password 'secret' you could write:
See https://en.wikipedia.org/wiki/Basic_authentication_scheme for details on the HTTP headers to include, and how to encode the user name and password. The "realm" can be set to any string, as the LabKey server does not support the creation of multiple basic authentication realms. The credentials provided can be a username & password combination or an API key, as described in more detail on this page.Additional details about HTTP authentication can be found here.
CSRF Token
Important: All mutating API actions (including insertRows, updateRows, and deleteRows) require a CSRF token in addition to user credentials. (For background and rationale, see Cross-Site Request Forgery (CSRF) Protection.) CSRF tokens are handled automatically by the client libraries, but code that invokes APIs via direct HTTP must obtain a CSRF token, send it with every API request. Follow these steps:
Send the "X-LABKEY-CSRF" cookie back to the server on every request. Note: Many HTTP libraries will re-send server cookies automatically.
Add an "X-LABKEY-CSRF" header with the value of the CSRF token to every request. Note: Many HTTP libraries have a mechanism for setting an HTTP header globally.
You can verify that your code is correctly handling CSRF tokens by invoking the test csrf action and ensuring a success response:
If you are looking for information on building a custom login page, see Modules: Custom Login Page.The following sections document the supported API actions in the current release of LabKey Server.For further examples of these actions in use, plus a tool for experimenting with "Get" and "Post" parameters, see Examples: Controller Actions / API Test Page
Query Controller API Actions
selectRows Action
The selectRows action may be used to obtain any data visible through LabKey’s standard query grid views.Example URL where "<MyServer>", "<MyProject>", and "<MyFolder>" are placeholders for your server, project, and folder names:
HTTP Method: GETParameters: Essentially, anything you see on a query string for an existing query grid is legal for this action.The following table describes the basic set of parameters.
Parameter
Description
schemaName
Name of a public schema.
query.queryName
Name of a valid query in the schema.
query.viewName
(Optional) Name of a valid custom grid view for the chosen queryName.
query.columns
(Optional) A comma-delimited list of column names to include in the results. You may refer to any column available in the query, as well as columns in related tables using the 'foreign-key/column' syntax (e.g., 'RelatedPeptide/Protein'). If not specified, the default set of visible columns will be returned.
query.maxRows
(Optional) Maximum number of rows to return (defaults to 100)
query.offset
(Optional) The row number at which results should begin. Use this with maxRows to get pages of results.
query.showAllRows
(Optional) Include this parameter, set to true, to get all rows for the specified query instead of a page of results at a time. By default, only a page of rows will be returned to the client, but you may include this parameter to get all the rows on the first request. If you include the query.showAllRows parameter, you should not include the query.maxRows nor the query.offset parameters. Reporting applications will typically set this parameter to true, while interactive user interfaces may use the query.maxRows and query.offset parameters to display only a page of results at a time.
query.sort
(Optional) Sort specification. This can be a comma-delimited list of column names, where each column may have an optional dash (-) before the name to indicate a descending sort.
query.<column-name>~<oper>=<value>
(Optional) Filter specification. You may supply multiple parameters of this type, and all filters will be combined using AND logic. The list of valid operators are as follows: eq = equals neq = not equals gt = greater-than gte = greater-than or equal-to lt = less-than lte = less-than or equal-to dateeq = date equal (visitdate~dateeq=2001-01-01 is equivalent to visitdate >= 2001-01-01:00:00:00 and visitdate < 2001-01-02:00:00:00) dateneq = date not equal neqornull = does not equal or is null isblank = is null isnonblank = is not null contains = contains doesnotcontain = does not contain startswith = starts with doesnotstartwith = does not start with in = equals one of a semi-colon delimited list of values ('a;b;c').
Response Format:The response can be parsed into an object using any one of the many JSON parsers available via https://www.json.org.The response object contains four top-level properties:
metaData
columnModel
rows
rowCount
metaData: This property contains type and lookup information about the columns in the resultset. It contains the following properties:
Property
Description
root
The name of the property containing rows. This is mainly for the Ext grid component.
totalProperty
The name of the top-level property containing the row count (in our case). This is mainly for the Ext grid component.
sortInfo
The sort specification in Ext grid terms. This contains two sub-properties, field and direction, which indicate the sort field and direction (“ASC” or “DESC”) respectively.
id
The name of the primary key column.
fields
an array of field information. name = name of the field type = JavaScript type name of the field lookup = if the field is a lookup, there will be three sub-properties listed under this property: schema, table, and column, which describe the schema, table, and display column of the lookup table (query).
columnModel: The columnModel contains information about how one may interact with the columns within a user interface. This format is generated to match the requirements of the Ext grid component. See Ext.grid.ColumnModel for further information.rows: This property contains an array of rows, each of which is a sub-element/object containing a property per column.rowCount: This property indicates the number of total rows that could be returned by the query, which may be more than the number of objects in the rows array if the client supplied a value for the query.maxRows or query.offset parameters. This value is useful for clients that wish to display paging UI, such as the Ext grid.
updateRows Action
The updateRows action allows clients to update rows in a list or user-defined schema. This action may not be used to update rows returned from queries to other LabKey module schemas (e.g., ms2, flow, etc). To interact with data from those modules, use API actions in their respective controllers.Example URL:
Content-Type Header: Because the post body is JSON and not HTML form values, you must include the 'Content-Type' HTTP header set to 'application/json' so that the server knows how to parse the incoming information.The schemaName and queryName properties should match a valid schema/query name, and the rows array may contain any number of rows. Each row must include its primary key value as one of the properties, otherwise, the update will fail.By default, all updates are transacted together (meaning that they all succeed or they all fail). To override this behavior, include a 'transacted': false property at the top level. If 'transacted' is set to 'false', updates are not automatic and partial updates may occur if an error occurs mid-transaction. For example, if some rows have been updated and an update produces an error, the rows that have been updated before the error will still be updated.The response from this action, as well as the insertRows and deleteRows actions, will contain JSON in the following format:
The response can be parsed into an object using any one of the many JSON parsers available via https://www.json.org.The response object will contain five properties:
schemaName
queryName
command
rowsAffected
rows
The schemaName and queryName properties will contain the same schema and query name the client passed in the HTTP request. The command property will be "update", "insert", or "delete" depending on the API called (see below). These properties are useful for matching requests to responses, as HTTP requests are typically processed asynchronously.The rowsAffected property will indicate the number of rows affected by the API action. This will typically be the same number of rows passed in the HTTP request.The rows property contains an array of row objects corresponding to the rows updated, inserted, or deleted, in the same order as the rows supplied in the request. However, the field values may have been modified by server-side logic, such as LabKey's automatic tracking feature (which automatically maintains columns with certain names, such as "Created", "CreatedBy", "Modified", "ModifiedBy", etc.), or database triggers and default expressions.
HTTP Method: POST (GET effective version 23.6)Content-Type Header: Because the post body is JSON and not HTML form values, you must include the 'Content-Type' HTTP header set to 'application/json' so that the server knows how to parse the incoming information.The post body for insertRows should look the same as updateRows, except that primary key values for new rows need not be supplied if the primary key columns are auto-increment.
HTTP Method: POSTContent-Type Header: Because the post body is JSON and not HTML form values, you must include the 'Content-Type' HTTP header set to 'application/json' so that the server knows how to parse the incoming information.The post body for deleteRows should look the same as updateRows, except that the client need only supply the primary key values for the row. All other row data will be ignored.
executeSql Action
This action allows clients to execute SQL.Example URL:
HTTP Method: POSTPost Body:The post body should be a JSON-encoded object with two properties: schemaName and sql. Example:
{ schemaName: 'study', sql: 'select MyDataset.foo, MyDataset.bar from MyDataset' }
The response comes back in exactly the same shape as the selectRows action, which is described at the beginning of the Query Controller API Actions section of this page.
Project Controller API Actions
getWebPart Action
The getWebPart action allows the client to obtain the HTML for any web part, suitable for placement into a <div> defined within the current HTML page.Example URL:
HTTP Method: GETParameters: The “webpart.name” parameter should be the name of a web part available within the specified container. Look at the Select Web Part drop-down menu for the valid form of any web part name.All other parameters will be passed to the chosen web part for configuration. For example, the Wiki web part can accept a “name” parameter, indicating the wiki page name to display. Note that this is the page name, not the page title (which is typically more verbose).
Assay Controller API Actions
assayList Action
The assayList action allows the client to obtain a list of assay definitions for a given folder. This list includes all assays visible to the folder, including those defined at the folder and project level.Example URL:
HTTP Method: POSTParameters: NonePost Body: NoneReturn value: Returns an array of assay definition descriptors.Each returned assay definition descriptor includes the following properties (this list is not exhaustive):
Property
Description
Name
String name of the assay, such as "Cell Culture"
protocolSchemaName
The full schema/type/assay name, such as "assay.General.Cell Culture"
id
Unique integer ID for the assay.
Type
String name of the assay type. "ELISpot", for example.
projectLevel
Boolean indicating whether this is a project-level assay.
description
String containing the assay description.
domainTypes
The domains included, such as Batch, Run, Result/Data, etc.
domains
An object mapping from String domain name to an array of domain property objects. (See below.)
Domain property objects include the following properties (this list is not exhaustive):
Property
Description
name
The String name of the property.
typeName
The String name of the type of the property. (Human readable.)
typeURI
The String URI uniquely identifying the property type. (Not human readable.)
label
The String property label.
description
The String property description.
formatString
The String format string applied to the property.
required
Boolean indicating whether a value is required for this property.
lookupContainer
If this property is a lookup, this contains the String path to the lookup container or null if the lookup in the same container. Undefined otherwise.
lookupSchema
If this property is a lookup, this contains the String name of the lookup schema. Undefined otherwise.
lookupQuery
If this property is a lookup, this contains the String name of the lookup query. Undefined otherwise.
Troubleshooting Tips
If you hit an error, here are a few "obvious" things to check:Spaces in Parameter Names. If the name of any parameter used in the URL contains a space, you will need to use "%20" or "+" instead of the space.Controller Names: "project" vs. "query" vs "assay." Make sure your URL uses the controller name appropriate for your chosen action. Different actions are provided by different controllers. For example, the "assay" controller provides the assay API actions while the "project" controller provides the web part APIs.Container Names: Different containers (projects and folders) provide different schemas, queries and grid views. Make sure to reference the correct container for your query (and thus your data) when executing an action.Capitalization: The parameters schemaName, queryName and viewName are case sensitive.
The HTTP Interface exposes a set of API endpoints that accept parameters & JSON payloads and return JSON results. These may be called from any program capable of making an HTTP request and decoding the JSON format used for responses (e.g., C++, C#, etc.). This page provides examples to help you get started using this interface.
Topics
The API Test Tool: Use the API Test Tool to perform HTTP "Get" and "Post" operations
Define a List: Design and populate a List for use in testing the Action APIs
Please note that only admins have access to the API Test Tool.To reach the test screen for the HTTP Interface, enter the following URL in your browser, substituting the name of your server for "<MyServer>" and the name of your container path for "<MyProject>/<MyFolder>"
Note that 'labkey' in this URL represents the default context path, but your server may be configured with a different context path, or no context path. This documentation assumes that 'labkey' (the default) is your server's context path.
Import a List for Testing Purposes
You will need a query table that can be used to exercise the HTTP Interface. In this section, we create and populate a list to use as our demo query table.
The updateRows action allows clients to update rows in a list or user-defined schema. This action may not be used to update rows returned from queries to other LabKey module schemas (e.g., ms2, flow, etc). To interact with data from those modules, use API actions in their respective controllers.Post Url:
The URL of Project Controller actions uses "project" instead of "query," in contrast to the Query Controller Actions described above.Lists. The Lists web part:
Assay List. Some web part names have spaces. You can find the valid form of web part names in the Select Web Part drop-down menu. A web part with a space in its name:
You can use the client-side language of your choice to access LabKey's HTTP Interface. This topic gives an example of using Perl.The callQuery.pl Perl script logs into a server and retrieves the contents of a list query called "i5397." It prints out the results decoded using JSON.Note that JSON 2.07 can be downloaded from http://search.cpan.org/~makamaka/JSON-2.07/ .Please use the callQuery.pl script attached to this page (click the link to download) in preference to copy/pasting the same script below. The wiki editor is known to improperly escape certain common perl characters. The code below is included for ease of reference only.
#!/usr/bin/perl -w use strict;
# Fetch some information from a LabKey server using the client API my $email = 'user@labkey.com'; my $password = 'mypassword';
use LWP::UserAgent; use HTTP::Request; my $ua = new LWP::UserAgent; $ua->agent("Perl API Client/1.0");
# Setup variables # schemaName should be the name of a valid schema. # The "lists" schema contains all lists created via the List module # queryName should be the name of a valid query within that schema. # For a list, the query name is the name of the list # project should be the folder path in which the data resides. # Use a forward slash to separate the path # host should be the domain name of your LabKey server # labkeyRoot should be the root of the LabKey web site # (if LabKey is installed on the root of the site, omit this from the url) my $schemaName="lists"; my $queryName="MyList"; my $project="MyProject/MyFolder/MySubFolder"; my $host="localhost:8080"; my $labkeyRoot = "labkey"; my $protocol="http";
#build the url to call the selectRows.api #for other APIs, see the example URLs in the HTTP Interface documentation at #https://www.labkey.org/Documentation/wiki-page.view?name=remoteAPIs my $url = "$protocol://$host/$labkeyRoot/query/$project/" . "selectRows.api?schemaName=$schemaName&query.queryName=$queryName";
#Fetch the actual data from the query my $request = HTTP::Request->new("GET" => $url); $request->authorization_basic($email, $password); my $response = $ua->request($request);
# use JSON 2.07 to decode the response: This can be downloaded from # http://search.cpan.org/~makamaka/JSON-2.07/ use JSON; my $json_obj = JSON->new->utf8->decode($response->content);
# the number of rows returned will be in the 'rowCount' propery print $json_obj->{rowCount} . " rows:n";
# and the rows array will be in the 'rows' property. foreach my $row(@{$json_obj->{rows}}){ #Results from this particular query have a "Key" and a "Value" print $row->{Key} . ":" . $row->{Value} . "n"; }
When you select External Tool Access from the (Username) menu, you will find resources for accessing LabKey from external tools. Depending on the features enabled on your server, you may see one or more of the following:
API Keys: Used to authenticate client code accessing LabKey Server using one of the LabKey Client APIs.
API Keys can be used to authenticate client code accessing LabKey Server using one of the LabKey Client APIs. Authentication with an API Key avoids needing to store your LabKey password or other credential information on the client machine. When desired, a single user can have multiple simultaneous API Keys that allow the same access for different purposes.
An API Key can be specified in .netrc/_netrc, provided to API functions, and used with external clients that support Basic authentication. Since a valid API Key provides complete access to your data and actions, it should be kept secret.
API Keys have security benefits over passwords:
They are tied to a specific server
They're usually configured to expire, and can be revoked by an administrator
They provide API access for users who sign in via single sign-on authentication mechanisms such as CAS and SAML
They also provide API access for servers configured for two-factor authentication (such as Duo or TOTP)
An administrator can configure the server to allow users to obtain an API Key (or token) once they have logged in. API Keys can be configured to expire after a duration specified by the administrator. An administrator also retains the power to immediately deactivate one or more API Keys whenever necessary.
In cases where access must be tied to the current browser session and run under the current context (e.g., your user, your authorizations and if applicable, your declared terms of use and PHI level, your current impersonation state, etc.), such as some compliance environments, you will need to use a session key. Session keys expire at the end of the session, whether by timeout or explicit logout.
Configure API Keys (Administrator)
Select > Site > Admin Console.
Under Configuration, click Site Settings.
Under Configure API Keys, check the box for Let users create API Keys.
Select when to Expire API Keys. Options:
Never (default)
7 days (1 week)
30 days (1 month)
90 days (3 months)
180 days (6 months)
365 days (1 year)
Click Save.
Access and Use an API Key (Developers/Users)
The API Key is a long, randomly generated token that provides an alternative authentication credential for use with APIs. A valid API Key provides complete access to your data and actions, so it should be kept secret.Once enabled, a logged-in user can retrieve an API Key via username > External Tool Access:Click Generate API Key to create one. In the popup, you can provide your own description of the usage of that key, which can help you later if you need to determine which key(s) may have expired. Click Generate API Key again.
Click the button to copy it to the clipboard. Important: the key itself will not be shown again and is not available for anyone to retrieve, including administrators. If you lose it, you will need to regenerate a new one.Click Done at the bottom of the page. Your key with any description will now be listed.You can then use this key in a .netrc/_netrc file or via clients that authenticate using Basic authentication. All access to the system will be subject to your authorization and logged with your user information.If needed, you can generate multiple API Keys and use them in different contexts at the same time to provide the same access under your credentials.
Note: When an administrator is impersonating a user, group or role, they cannot generate an API Key.
Example: .netrc/_netrc File
To avoid embedding credentials into your code, you can use the API Key as a password within a .netrc/_netrc file. When doing so, the username is "apikey" (instead of your email address) and the password is the entire API Key. This is the recommended method of using an API Key; it is compatible with all LabKey client libraries.
Any API use via a LabKey client library will be able to access the server with your permissions, until the key expires or is terminated by an administrator.
Troubleshooting
If you see an error like one of these:
Error in handleError(response, haltOnError) : HTTP request was unsuccessful. Status code = 401, Error message = User does not have permission to perform this operation.
labkey.exceptions.RequestAuthorizationError: '401: User does not have permission to perform this operation.'
Check to see if you are using an invalid API Key, either one that has expired, been revoked, or has additional characters (such as the "apikey|" prefix that was previously used with API Keys and is no longer used). The invalid key could be in your script or in the .netrc/_netrc file.
Manage API Keys (Administrator)
A site administrator can manage API Keys generated on the server using the APIKey query. Link to it from the top of the null username > External Tool Settings page.You will see the keys that have been generated on this server, listed by username and displaying the time of creation as well as expiration (where applicable), last usage, and a description if one was included. Note that session keys are not listed here.To revoke an API Key, such as in a case where it has been compromised or shared, select the row and click (Delete). To revoke all API Keys, select all rows and delete.
Premium Feature — Available in the Professional and Enterprise Editions of LabKey Server. Learn more or contact LabKey.
External applications can dynamically query the data stored in LabKey in order to create reports and perform analyses.LabKey Server implements the Postgres network wire protocol and emulates the database's responses, allowing supported tools to connect via standard Postgres ODBC and JDBC drivers. Clients connect to LabKey Server, not the backend database, ensuring consistent handling of data and standard LabKey user permissions.This enables users to continue outside tools they may already be using with data stored in LabKey.
Your server must include the connectors module to support external analytics connections. This module leverages a selected port to pass data from LabKey to an external tool. Across this connection, the user will have access to the data they could also access directly from the UI or via API. The connection does not provide direct access to the database or to any schemas or tables that would not ordinarily be accessible.To enable connections, follow the instructions below.
Select > Site > Admin Console.
Under Premium Features, click External Analytics Connections.
On the page Enable External Analytics Connections, place a check mark next to Allow Connections.
By default the server will listen for client requests on port 5435. If desired, you can change the port number within the range: 1 to 65535.
You'll see the Host Name displayed. If necessary, you can check Override Host Name to use another URL.
The Require TLS settings on this page are described in the next section.
Click Save.
Override Host Name
The Host Name is shown on the External Tool Access pages for users. ODBC connections use this value as the "Server" for the connection. The default value is the base server URL with "-odbc" included, but can be customized as needed. The value provided here should not include a scheme (e.g., http) or port.
Secure ODBC Using TLS
Encrypting ODBC connections using Transport Level Security (TLS) is recommended for all deployments and are supported for both cloud and on-premise servers. Secure ODBC connections can piggyback Tomcat's TLS configuration (both certificates and keys), or the server can automatically generate a key pair and self-signed certificate. This topic describes how an administrator configures LabKey Server to accept these secure connections.To require that ODBC connections are encrypted, you can check the box for Require TLS for encrypted connections. In this section you learn how to configure the LabKey Server deployment. LabKey Server can either get its configuration from Tomcat's HTTPS settings, or use a self-signed certificate. The page also shows the expiration dates for the available certificates.
Use LabKey's HTTPS Certificate
When Tomcat's HTTPS Certificate is chosen, LabKey Server uses the same settings and certificate for ODBC connections as for other TLS/SSL configuration. Learn more in this topic:
Note that while generally both commas or colons are allowed as the delimiter for the ciphers list, to make it work with ODBC connections, a colon delimiter must be used in separating cipher suites. For example:
When Self-signed certificate (managed by LabKey Server) is chosen:
Choose the validity period for newly generated certificates. The server will automatically generate a new certificate when the old one expires.
Check the box to Regenerate certificate to force the server to generate a new certificate immediately.
Click the (download) link to download the certificate. This link will be active for the option selected by radio button.
Click Save.
Configure PostgreSQL Client for Secure Connections
When you configure the client machine where your external tools will run, follow the guidance in the topic for your operating system, making note of the data source name (DSN) requirements for ensuring the connection is secure.
Users can review the settings you have configured by selecting (username) > External Tool Access. Values on this page will assist in configuring their local connection; tooltips provide additional guidance. Administrators can use links to download drivers for installation and users can download the certificate as needed.
Additional Topics for External Tools
Configure ODBC Access from Host Machine: Set up the host machine to use the ODBC connection, including installation of the TLS access certificate for each user.
Premium Feature — Available in the Professional and Enterprise Editions of LabKey Server. Learn more or contact LabKey.
An ODBC (Open Database Connectivity) Connection exposes the LabKey schema and queries as a data source to external clients for analysis and reporting. This enables users to continue outside tools they may already be using with data stored in LabKey.This topic covers how to set up the host Windows machine to use the ODBC connection, including installation of the TLS access certificate for each user. An administrator must already have enabled ODBC connections on LabKey Server following this topic:
To confirm that configuration has been completed, and obtain the relevant settings to use when setting up your driver, users can select (username) > External Tool Access.If you will be using a secure ODBC connection, you will need to download and install the certificate from this page and install it on your local machine.
Windows: Install PostgreSQL Driver
On the client machine, install the latest release of the PostgreSQL ODBC driver.
Find and download the latest release available for your system.
There are 32-bit and 64-bit drivers available. You will need to install the version that matches your client tool, not necessarily the host machine. For example, if you have a 32-bit version of Excel, then install the 32-bit ODBC driver, even if you have a 64-bit machine.To cover all options, you can install both versions.
Windows: Create a Data Source Name (DSN)
On the client machine, create a "data source name" (DSN) to wrap a data container on LabKey Server. Creating a "system" DSN, as shown below, makes it available to various clients. Client tools use the ODBC driver to query this DSN.
On Windows, open the ODBC Data Source Administrator.
Search for it as ODBC Data Sources (##-bit) under Windows Administrative Tools.
Choose "Run as Administrator".
Click the System DSN tab.
Click Add....
Select PostgreSQL Unicode from the list. This is the driver you installed above.
Click Finish.
Enter the configuration details in the popup:
Data Source: This is the name you will select within the client tool. If you will need different data source configurations for different tools or different LabKey container paths, differentiate them by name of the data source.
Description: This can be any text.
Database: A LabKey container path, that is, the project or folder you are connecting to. Include a leading slash in the path, for example, "/Home" or "/Home/MyDataFolder". Be sure there are no leading or trailing spaces in this path.
Server: The server you are connecting to, as listed in the External Tool Access grid. For example, www.labkey.org or localhost. Note that for cloud instances of LabKey Server, this URL may differ from the actual server URL; i.e. if your server is lk.myserver.net, the ODBC URL might be lk.myserver-odbc.net.
Port: This number must match the port enabled on the server. 5435 is the default used by LabKey Server.
User Name: The user this connection will authenticate against. This user should have at least the Reader role in the LabKey Server container.
Password: The password for the above user.
Click Test to ensure the connection is successful.
Click Save to finish.
Configure PostgreSQL Client for Secure Connections
PostgreSQL supports the following TLS connection modes. When secure connections are enforced through LabKey Server, connections through disable and allow modes are not successful.
disable
allow
prefer
require
verify-ca
verify-full
For details on these modes see the PostgreSQL documentation at Protection Provided in Different Modes .For modes verify-ca and verify-full, users will need to place the certificate for the server in the location specified in the PostgreSQL docs at Client Verification of Server Certificates If the client has been configured to trust the certificate (by adding it to the CA list) verify-ca will also work.
Install Certificate on Windows
To use the secure ODBC connection, the user must install the self-signed certificate on their own Windows machine as follows:
First, if you are using an existing certificate, it is good practice to back it up before making these changes.
Go to the directory where your driver will expect to find it (creating the "postgresql" subdirectory if it is not already present). Similar to:
C:\Users\username\AppData\Roaming\postgresql
If there is a root.crt file, make a backup copy of it (such as backupRoot.crt).
You will append the new certificate to the end of the original root.crt file.
Obtain the new certificate from within LabKey Server:
From the (user) menu, select External Tool Access.
You will see the current ODBC access settings. Click Download Certificate to download the certificate.
In the same directory as identified above, (i.e. C:\Users\username\AppData\Roaming\postgresql) append the new certificate to your existing root.crt file, or create a new root.crt file to contain it.
More instructions about obtaining and placing the certificate are available here:
Premium Feature — Available in the Professional and Enterprise Editions of LabKey Server. Learn more or contact LabKey.
An ODBC (Open Database Connectivity) Connection exposes the LabKey schema and queries as a data source to external clients for analysis and reporting. This enables users to continue outside tools they may already be using with data stored in LabKey.
Note: At this time, only commercial ODBC drivers can be used for using Excel from OSX. Learn more in this support topic on the Microsoft support site:
To confirm that configuration has been completed, and obtain the relevant settings to use when setting up your driver, users can select (username) > External Tool Access.If you will be using a secure ODBC connection, you will need to download and install the certificate from this page and install it on your local machine.
Obtain and Install an ODBC Driver
While the free PostgreSQL ODBC driver can successfully be used from Windows, you must currently use a commercial ODBC driver on a Mac.
Follow the instructions provided with the driver you choose to configure and define the connection. You will define a Data Source Name (DSN), typically providing information like the server, port, "database" (i.e. the container path within LabKey Server), user/password, etc. The method may vary with different commercial drivers, and the names of fields can differ. Values provided in the ODBC Access panel of the LabKey UI will help you find the values to use here.For example, depending on the driver, you might see and test settings from a Connection tab as shown here:
Access Data from External Tools
To access your LabKey data from supported external tools, you'll configure the tool to read the data across the DSN you configured. The specific menu options to use will vary by tool.Learn more about specific supported integrations in this topic:
Premium Feature — Available in the Professional and Enterprise Editions of LabKey Server. Learn more or contact LabKey.
An ODBC (Open Database Connectivity) Connection exposes the LabKey schema and queries as a data source to external clients for analysis and reporting. This topic outlines using an ODBC connection with those tools:
Step 1: Before you can set up any of the tools covered in this topic, an administrator must have configured LabKey Server to accept ODBC connections as covered in this topic:
Step 2: Your local client machine must also have been setup following the directions in this topic, written for a Windows client. A similar process will need to be followed for other client machines
Windows: Configure ODBC Access: Set up the host machine to use the ODBC connection, including installation of the TLS access certificate for each user and definition of the DSN (Data Source Name) the external tool will use.
Step 3: Configure the external tool to use that data source, as described in this topic for several common tools. Other tools offering ODBC support may also be able to connect using similar steps.
The underlying exposure mechanism is an implementation of the PostgreSQL wire protocol. Each LabKey container (a project or folder) is surfaced to clients as a separate PostgreSQL "database". These "databases" expose the LabKey virtual schema (the same view of the data provided by the Query Schema Browser).Queries through an ODBC connection respect all of the security settings present on the LabKey Server container. Clients must have at least the Reader role to access/query the data. Only read access is supported; data cannot be inserted or updated using the virtual schema over an ODBC connection.
Tableau Desktop
To load data into Tableau Desktop:
In Tableau Desktop, go to Data > New Data Source > More… > Other Databases (ODBC).
Place a checkmark next to DSN and select your DSN in the dropdown. Click Connect.
The Database field in the data source configuration should contain the container path to the project and folder on the LabKey Server. Shown below, the project named "Testing".
Search for and select the Schema - 'core' is shown in the screenshot below.
Search for and select the Table - 'Modules' is shown in the screenshot below.
We recommend that you set the Connection to "Extract" instead of "Live". (This helps to avoid the following errors from the ODBC driver: "ODBC escape convert error".)
Excel
To load data into Excel:
In Excel, open an empty sheet and click the Data tab
Select Get Data > From Other Sources > From ODBC. (Note this path may vary, depending on your version of Excel.)
In the From ODBC popup dialog, select the system Data Source Name (DSN) you created; shown below it is named "MyLabKeyData". This DSN includes both the path to the intended container and credential to use.
You can optionally click Advanced Options and enter a SQL query now.
Click OK/Next.
Note that if you are asked to reenter your credentials when you click Next, you can select the Windows tab to have the option to Use my current credentials (i.e. those assigned in the DSN).
Click Connect to continue to the Navigator dialog where you can select a table to open or directly see the results of the query you specified.
If you chose not to provide a SQL query, select the table to load using the Navigator dialog. Select the desired table and click Load.
The data will be selected from the server and loaded into the worksheet.
Controlling Excel Data Loading
To control the SQL SELECT statement used by Excel to get the data, such as adding a WHERE or JOIN clause, double-click the table/query in the Queries and Connections panel. In the Power Query Editor, click Advanced Editor.To control the refresh behavior, go to Data tab > Connections > Properties. The Refresh control panel provides various options, such as refreshing when the sheet is opened.Note that saving a sheet creates a snapshot of the data locally. Use with caution if you are working with PHI or otherwise sensitive data.
Access
Access can be run in snapshot or dynamic modes. Loading data into Access also provides a path to processing in Visual Basic.See the Microsoft documentation at Add an ODBC Data Source
SQL Server Reporting Services (SSRS)
SQL Server Reporting Services is used for creating, publishing, and managing reports, and delivering them to the right users in different ways, whether that's viewing them in a web browser, on their mobile device, or via email.To use SSRS, you must install both the 32-bit and 64-bit ODBC drivers on your Windows machine. Visual Studio and Report Builder are 32 bit and recent versions of SSRS are 64 bit. Important: give both the 32-bit and 64-bit ODBC data sources the same name.Setting up both the Reporting Services and Visual Studio are covered in this topic:
Open the Database Explorer app. (If you don't see it, install the Database Toolbox.)
In the Database Explorer, click New Query.
In the Connect to a Data Source select your DSN and provide a username and password.
In the popup dialog, select the target Schema. The Catalog (a LabKey container) is determined by the DSN.
Select a table to generate SQL statements.
JMP
JMP is statistical analysis software with a flexible visual interface. You configure JMP to use existing local Data Sources on your machine, and can also define a new DSN, perhaps with a different container path, directly from within JMP.
Install and open JMP.
From the File menu, choose Database (updated), then Open Table...
Click New Connection...
Switch to the Machine Data Source tab and choose the Data Source Name to use.
You could also define a new one from here by clicking New...
Click OK.
On the next screen, confirm the connection is to the expected "Database" (container path on LabKey Server). You can update the credential here if desired.
Click OK.
You can now select a schema and table, then click Open Table. The table will open into a JMP data table.
Or instead of opening a table directly, click Advanced... and construct a query.
PowerBI
Connecting from a desktop installation of PowerBI to data managed by LabKey is supported.
When you open PowerBI, click Get Data.
Choose Get Data from Another Source.
In the popup, click Other, then choose ODBC.
Click Connect.
Select the desired Data source name (DSN) and click OK.
Note that if you are asked to reenter your credentials when you click Next, you can select the Windows tab to have the option to Use my current credentials (i.e. those assigned in the DSN). This setting will be saved for future uses of this DSN in PowerBI.
Select the desired data using the Navigator and Load for your analysis.
Note that server-based installations of PowerBI, such as via AWS, may experience timeouts and inconsistent behavior due to gateway interactions. Only the desktop version of PowerBI is considered supported.
DBeaver
DBeaver can connect using the Postgres JDBC driver. However, to be compatible, it must be configured as a Generic JDBC driver.
You can now browse the schema and run SQL queries.
Other Tools
These other external tools have not been extensively tested and are not officially supported, but have been reported to be compatible with LabKey using ODBC connections.
If your ODBC connection and external tool are not working as expected, return to the setup instructions to confirm proper configuration. In particular, you may need to update the Postgres ODBC driver to the latest version and confirm that it is compatible with the tool you are using.
Error Messages (may vary by tool)
Possible cause(s) and solutions
ERROR: database path … could not be resolved to an existing container
Check that the path ("Database" in your Data Source Name configuration) exists, and that your credential has access to it. Confirm that there are no leading or trailing spaces in the path in your DSN configuration.
Password authentication failed
Check that the password is correct. This error also occurs when your DSN configuration does not include a port number.
Connection refused Is the server running on host "hostname.com" and accepting TCP/IP connections on port 5435?
Ensure that the LabKey Server has enabled ODBC connections, that you have the right host name and port, and that firewalls or other network configurations are not blocking TCP network traffic.
ODBC: ERROR [IM014] The specified DSN contains an architecture mismatch between the driver (32-bit) and Application (64-bit)
Either a driver of the wrong architecture is in use, or no driver is installed on the local machine.
DataSource Error: ODBC: ERROR [HY000] Error while executing the query
Check the server logs; this error may indicate a query with a syntax error or other problem further described in the log. You may also see this error if you need to update the driver.
SSL error: certificate verify failed
Your client tool is attempting to validate the server's certificate and does not trust it. Try the "require" option for the SSL Mode, and ensure that the server's certificate has been added to your root.crt file.
SSL error: unsupported protocol
The server and client failed to negotiate a TLS protocol. Ask your server admin to check the protocol and ciphers setting, and validate that the configured keystore is valid. Or switch the LabKey Server ODBC configuration to use the self-signed certificate option.
Failure to Decrypt Credentials. The data source credentials could not be decrypted. Please restart the program. If you continue to see this message try resetting permissions.
This error (or similar) may be raised after a Windows upgrade or other machine maintenance. Try reinstalling the ODBC driver and redefining your DSN.
Debug with Loggers
To get additional detail about ODBC connections, you can temporarily turn on DEBUG logging for the following loggers:
org.labkey.connectors.postgresql.SocketListener : listener for incoming ODBC connections. This logger will record new connections and the port accessed (whether or not said connections are actually successful).
org.labkey.connectors.postgresql.SocketConnection : See more detail about packets, protocols, and queries. This logger will record credential authentication issues, "Database" path accessibility issues, and other details that may point to details to correct.
For example, if you use a "Database" path that exists, but includes a trailing space, the connection will fail, but the client error reported may collapse such white space. Using debug logging you can see the issue more clearly (in the quoted string):
DEBUG SocketListener 2022-02-09T08:45:36,304 PostgreSQL Listener : New connection for Postgres socket 5435 DEBUG SocketConnection 2022-02-09T08:45:36,324 Postgres wire protocol 4 : First packet protocol version: 1234.5679 DEBUG SocketConnection 2022-02-09T08:45:36,327 Postgres wire protocol 4 : Protocols set in server config - TLSv1.2 DEBUG SocketConnection 2022-02-09T08:45:36,351 Postgres wire protocol 4 : First packet protocol version: 3.0 DEBUG SocketConnection 2022-02-09T08:45:36,351 Postgres wire protocol 4 : Properties: {database=/Tutorials/HIV Study , user=[username]} DEBUG SocketConnection 2022-02-09T08:45:36,351 Postgres wire protocol 4 : Sending error to client: database path "/Tutorials/HIV Study " could not be resolved to an existing container
Tableau, Date Fields, and Escape Convert Errors
If you are working with date fields in Tableau Desktop and see an error like Error message: "Bad Connection..."
Bad Connection: Tableau could not connect to the data source. ODBC escape convert error
<snip>Generated SQL statement is shown here</snip>
Edit the DSN you are using for connecting Tableau Desktop and confirm you selected "Extract". See above for a screenshot.
Premium Feature — Available in the Professional and Enterprise Editions of LabKey Server. Learn more or contact LabKey.
This topic explains how to set up SSRS to create reports and perform analyses on data stored in LabKey Server. It assumes that you have previously set up an ODBC Connection to LabKey Server and configured your Windows machine.
Go to Report Server Configuration Manager, select Server Name and Report Server Instance: SSRS → Connect
In the Reporting Services Configuration Manager, ensure your screens look similar to below:
Report Server Status:
b) Service Account:
c) WebService URL & Web Portal URL (should be clickable):
d) Database (Change Database or Change Credentials to set these values for the first time):
e) You may need to setup an Execution Account.
The Account will request a format like <Domain><Username>, there is no domain on the LabKey Server so leave that part blank.
Use the same account as your ODBC data sources (i.e your LabKey user credentials)
Visual Studio
In Visual Studio create a Report Server Project
To set up the data source, add a new Shared Data Source
Set type to ODBC
The Connection string just needs to show which DSN to use. This should be the name that you used for both your 32 bit and 64 bit data sources.
Click Credentials and select Do not use credentials
Right click on Project → Properties → set properties as shown below:
You should now be able to use this datasource to create datasets and reports. Your report should work in preview mode and when deployed to the report server.
Notes
The Query Designer is an optional UI to aid in the creation of report Datasets. Since it is generating and parsing SQL queries using common SQL languages like Transact-SQL, some of the enhanced features of LabKey SQL cannot be generated or parsed using the Query Designer.
LabKey pivot queries must be manually entered and the Query Designer cannot be used on a LabKey pivot query.
The Table Query Designer (available when Dataset Query Type is Table) is not available when using an ODBC data source, instead the Text Query Designer will be shown. This is a limitation of the .Net ODBC controller.
Parameters in Queries. When wanting to use parameters in Dataset queries, ‘?’ should be used in your query to insert your parameter. Then in the Parameters section you can define the name of the parameters used in the report UI. The parameters will be listed in the order they appear in the query.
To enable programmatic use of data as if "attached" to a given session, an administrator can configure the server to let users obtain a Session Key (or token) once they have logged in via the web UI. This key can be used to authorize client code accessing LabKey Server using one of the LabKey Client APIs. Using any API Key avoids copying and storing your credentials on the client machine. If necessary, users can generate multiple Session Keys for the same session.
As an application example, regulatory compliance may impose stringent data access requirements, such as having the user declare their intended use of the data, provide their IRB number and necessary PHI level, and sign associated terms of use documents every time they log in. This information is logged with each access of the data for later review or audit.When using a Session Key, access is tied to a given user's current browser session and runs under the current context (e.g., your user, your authorizations and if applicable, your declared terms of use and PHI level, your current impersonation state, etc.) then expires at the end of the session, whether by timeout or explicit logout.
Enable Session Keys
Select > Site > Admin Console.
Under Configuration, click Site Settings.
Under Configure API Keys, check Let users create session keys.
Click Save.
Obtain and Use a Session Key
Once enabled, the user can log in, providing all the necessary compliance information, then retrieve their unique Session Key from the username > External Tool Access menu:
Click Generate Session Key. The Session Key is a long, randomly generated token that is valid for only this single browser session.
Click the button to copy it to the clipboard.
Important: the key itself will not be shown again and is not available for anyone to retrieve, including administrators. If you lose it, you will need to regenerate a new one.
Click Done at the bottom of the page.
You can then paste this key into a script, tying that code's authorization to the browser session where the key was generated. The Session Key can also be used in a netrc file or via an external client that supports Basic authentication, as shown in API Keys. When using a Session Key within a netrc file, you use the login "apikey." When using a Session Key, the code's actions and access will be logged with your user information and assertions you made at login time.
Example: netrc File
To avoid embedding credentials into your code, you can use a Session Key as a password within a .netrc/_netrc file. When doing so, the username is "apikey" and the password is the entire Session Key including the prefix.
Modules encapsulate functionality, packaging resources together for simple deployment within LabKey Server. Modules are developed by incrementally adding resources as files within a standardized directory structure. For deployment, the structure is archived as a .module file (a standard .zip file renamed with a custom file extension).
A wide variety of resources can be used, including but not limited to: R reports, SQL queries and scripts, API-driven HTML pages, CSS, JavaScript, images, custom web parts, XML assay definitions, and compiled Java code. A module that doesn't contain any Java code is called a file-based module and enables custom development without compiling, letting you directly deploy and test module resources, oftentimes without restarting the server.
This topic helps module developers extend LabKey Server. To learn about pre-existing features built into LabKey Server's existing modules, see LabKey Modules.
A module that includes queries, reports, and/or views directories. Create file-based SQL queries, reports, views, web parts, and HTML/JavaScript client-side applications. No Java code required, though you can easily evolve your work into a Java module if needed.
Modules containing Java code require a build/compile step before deployment. File-based modules (that do not contain Java) do not need compilation. Most module functionality can be accomplished without the need for Java code, including "CRUD" applications (Create-Retrieve-Update-Delete applications) that provide views and reports on data on the server, and provide some way for users to interact with the data. These applications will typically use some combination of client APIs like: selectRows, insertRows, updateRows, etc. Learn more about the APIs available here: LabKey Client APIs.Other advanced custom functionality, including defining new assay types, working with the security API, and manipulating studies, can also generally be accomplished with a file-based module.To create your own server actions (i.e., code that runs on the server, not in the client), Java is generally required. Trigger scripts, which run on the server, are an exception: trigger scripts are a powerful feature, sufficient in many cases to avoid the need for Java code.As you develop, keep in mind that both client- and server-side APIs may change over time as the LabKey Server code base evolves; custom modules may require changes to keep them up to date.
Set Up for Module Development
Use the following topic to set up a development machine for building and compiling (when necessary) LabKey modules: Set Up a Development Machine
Premium Features AvailableSubscribers to the Professional or Enterprise Edition of LabKey Server can load modules on production servers without starting and stopping them, and on development machines, can edit module resources from within the user interface. Learn more in these topics:
The topics below show you how to create a module, how to develop the various resources within the module, and how to package and deploy it to LabKey Server.
Map of Module Files - An overview of the features and file types you can include in modules.
Example Modules - Representative example modules from the LabKey Server open source project.
LabKey Server's functionality is packaged inside of modules. For example, the query module handles the communication with the databases, the wiki module renders Wiki/HTML pages in the browser, the assay module captures and manages assay data, etc.You can extend the functionality of the server by adding your own module. Here is a partial list of things you can do with a module:
Create a new assay type to capture data from a new instrument.
Add a new set of tables and relationships (= a schema) to the database by running a SQL script.
Develop file-based SQL queries, R reports, and HTML views.
Build a sequence of scripts that process data and finally insert it into the database.
Define novel folder types and web part layouts.
Set up Extract-Transform-Load (ETL) processes to move data between databases.
Modules provide an easy way to distribute and deploy code to other servers, because they are packaged as single .module files, really just a renamed .zip file. When the server detects a new .module file, it automatically unzips it, and deploys the module resources to the server. In many cases, no server restart is required. Also, no compilation is necessary, assuming the module does not contain Java code or JSP pages.The following tutorial shows you how to create your own "Hello World" module and deploy it to a local testing/development server.
In this step you will set up a test/development machine, which compiles LabKey Server from its source code.If you already have a working build of the server, you can skip this step.
Download the server source code and complete an initial build of the server by completing the steps in the following topic: Set Up a Development Machine
Add the following property/value pairs to module.properties. This is a minimal list of properties needed for deployment and testing. You can add a more complete list of properties later on if desired. Find available properties in: module.properties Reference.
Add a file named "build.gradle" to the root directory of your module, resulting in: <LK_ENLISTMENT>/server/modules/helloworld/build.gradle
Add these lines to the file build.gradle:
plugins { id 'org.labkey.build.fileModule' }
Note that if you are generally building your server from artifacts, using "buildFromSource=false" in your enlistment, you should also add a text file "<LK_ENLISTMENT>/server/modules/helloworld/gradle.properties" that contains the line: "buildFromSource=true". Otherwise your build will fail to "find the artifact" for this local module.
Confirm that your module will be included in the build:
Open a command window.
Go to the <LK_ENLISTMENT> directory.
Call the Gradle task:
gradlew projects
In the list of projects, you should see the following (other lines not shown):
Confirm that HelloWorld has been deployed to the server by going to > Site > Admin Console. Click the Module Information tab. Open the node HelloWorld to see properties.
Add a Default Page
begin.htmlEach module has a default home page called "begin.view". In this step we will add this page to our module. The server interprets your module resources based on a fixed directory structure. By reading the directory structure and the files inside, the server knows their intended functionality. For example, if the module contains a directory named "assays", this tells the server to look for XML files that define a new assay type. Below, we will create a "views" directory, telling the server to look for HTML and XML files that define new pages and web parts.
Inside helloworld, create a directory named "resources".
Inside resources, create a directory named "views".
Inside views, create a file named "begin.html". (This is the default page for any module.)
Open begin.html in a text editor, and add the following HTML code:
<p>Hello, World!</p>
Test the Module
Stop the server.
Build the server using 'gradlew deployApp'.
Restart the server.
Navigate to a test folder on your server. A "Tutorials" project is a good choice.
Confirm that the view has been deployed to the server by going to > Go to Module > HelloWorld.
The begin.html file you created is displayed as a panel like the following.
Notice the URL ends with "HelloWorld-begin.view". Learn more about using the URL to access module resources in this topic: LabKey URLs.
If you don't see the HelloWorld module listed, be sure that you rebuilt your server after adding the view.
Enable the module in your folder as follows:
Go to > Folder > Management and click the Folder Type tab.
In the list of modules on the right, place a checkmark next to HelloWorld.
Click Update Folder.
Modify the View with Metadata
You can control how a view is displayed by using a [ViewName].view.xml metadata file. For example, you can define the title, framing, and required permissions.begin.view.xml
Add a file to the views directory named "begin.view.xml". Note that this file has the same name (minus the file extension) as begin.html: this tells the server to apply the metadata in begin.view.xml to begin.html:
Add the following XML to begin.view.xml. This tells the server to: display the title 'Begin View', display the HTML without the default framing that looks like a web part.
You may need to stop, rebuild, restart your server and return to the module to see the result.
The begin view now looks like the following:
Experiment with other possible values for the 'frame' attribute:
portal (If no value is provided, the default is 'portal', i.e. show the view like a web part.)
title
dialog
div
left_navigation
none
For the next step of the tutorial, if the 'frame' is none, you will only see a web part frame in page admin mode. To show the frame to all users, set the 'frame' to 'portal' before continuing.
Hello World Web Part
You can also package your custom view as a web part using another metadata file.begin.webpart.xml
In the helloworld/resources/views directory add a file named "begin.webpart.xml". This tells the server to surface the view inside a web part. Your module now has the following structure:
<webpart xmlns="http://labkey.org/data/xml/webpart" title="Hello World Web Part"> <view name="begin"/> </webpart>
Save the file.
Return to your test folder using the project menu in the upper left.
Enable the Hello World module in your folder, if you did not do so earlier:
Go to > Folder > Management and click the Folder Type tab.
In the list of modules on the right, place a checkmark next to HelloWorld.
Click Update Folder.
In your test folder, enter > Page Admin Mode, then click the pulldown menu that will appear in the lower left: <Select Web Part>.
Select the web part Hello World Web Part and click Add.
Your new web part will be added to the page:
Click Exit Admin Mode.
Note that if you left the begin.view.xml attribute 'frame' set to 'none', the frame is now gone. If you want to see web part framing for your new part, edit the begin.view.xml to set the 'frame' to 'portal' before continuing.
Hello World Module View
The final step enhances this view using the JavaScript API to display the name of the user and list the modules deployed on this server.
Open begin.html and replace the HTML with the content below.
var qwp1 = new LABKEY.QueryWebPart({ renderTo: 'queryDiv', title: 'LabKey Modules', schemaName: 'core', queryName: 'Modules', }); </script>
Refresh the server to see the changes.
Note that once the module structure is defined, you can interate and test some changes directly in the built deployment under <LK_ENLISTMENT>/build/deploy/modules/helloworld, letting you avoid rebuilding for each change. Once you are happy with the results, be sure that any changes you make in this way are migrated back to the <LK_ENLISTMENT>/server/modules/helloworld source location so that your next module build includes them.
Once you've refreshed, the web part will look something like this, displaying your own server's modules:
Make a .module File
You can distribute and deploy a module to a production server by making a helloworld.module file (a renamed .zip file).
Be sure that any module edits you made under the build location were migrated back to the source under <LK_ENLISTMENT>/server/modules/helloworld.
In anticipation of deploying the module on a production server, add the property 'BuildType: Production' to the module.properties file:
module.properties
Name: HelloWorld ModuleClass: org.labkey.api.module.SimpleModule BuildType: Production
Rebuild the module by running:
gradlew :server:modules:helloworld:deployModule
The build process creates a helloworld.module file at:
This file can be deployed by copying it to a production server's externalModules directory, or by deploying it through the server UI. When the server detects changes in the externalModules directory, it will automatically unzip the .module file and deploy it. You may need to restart the server to fully deploy the module.
Important: While you built this module in your development machine's "modules" location, Do not place your own modules in the "modules" directory of a server run as a service (locally or otherwise), as it will be overwritten during upgrades.Here is a completed helloworld module: helloworld.module
The following directory structure follows the pattern for modules as they are checked into source control. The structure of the module as deployed to the server is somewhat different, for details see below. If your module contains Java code or Java Server Pages (JSPs), you will need to compile it before it can be deployed.Items shown in lowercase are literal values that should be preserved in the directory structure; items shown in UPPERCASE should be replaced with values that reflect the nature of your project.
MODULE_NAME
│ module.properties docs
└──resources
│ module.xml docs, example
├───assay docs
├───credits docs
├───domain-templates docs
├───etls docs
├───folderTypes docs
├───olap example
├───pipeline docs, example
├───queries docs
│ └───SCHEMA_NAME
│ │ QUERY_NAME.js docs, example (trigger script)
│ │ QUERY_NAME.query.xml docs, example
│ │ QUERY_NAME.sql example
│ └───QUERY_NAME
│ VIEW_NAME.qview.xml docs, example
├───reports docs
│ └───schemas
│ └───SCHEMA_NAME
│ └───QUERY_NAME
│ MyRScript.r example
│ MyRScript.report.xml docs, example
│ MyRScript.rhtml docs
│ MyRScript.rmd docs
├───schemas docs
│ │ SCHEMA_NAME.xml example
│ └───dbscripts
│ ├───postgresql
│ │ SCHEMA_NAME-X.XX-Y.YY.sql example
│ └───sqlserver
│ SCHEMA_NAME-X.XX-Y.YY.sql example
├───scripts docs, example
├───views docs
│ VIEW_NAME.html example
│ VIEW_NAME.view.xml example
│ TITLE.webpart.xml example
└───web docs
└───MODULE_NAME
SomeImage.jpg
somelib.lib.xml
SomeScript.js example
Module Layout - As Source
If you are developing your module inside the LabKey Server source, use the following layout.
The standard build targets will automatically assemble the directories for deployment. In particular, the standard build target makes the following changes to the module layout:
Moves the contents of /resources one level up into /mymodule.
Uses module.properties to create the file config/module.xml via string replacement into an XML template file.
Compiles the Java /src dir into the /lib directory.
Premium Feature — This feature is available with the Professional and Enterprise Editions of LabKey Server. It requires the addition of the moduleEditor module to your distribution. Learn more or contact LabKey.
With the moduleEditor module, file-based modules (i.e. modules that do not contain Java or other code that requires compilation) can be loaded and updated on a production server using the user interface without needing to start and stop the server. This feature makes it easier to update queries, reports, and views that can be maintained through a file based module.
Create a file based module containing the resources needed. Typically you would create your module on a development machine, and build it, generating a file named [MODULE_NAME].module. To load the content via this process, the name of the .module file needs to be all lower case.If you would like a small module to test this feature as shown in the walkthrough on this page, download:
On your LabKey Server, add your new module by name as follows. You must be a site admin or have the Platform Developer role to add modules, and this feature is only supported for servers running in production mode.
Select > Site > Admin Console.
Click the Module Information tab.
Click Module Details above the list of current modules.
Click Create New Empty Module.
Enter the name of your new module (mixed case is fine here) and click Submit.
You will now see your module listed. Notice the module type shown is "SimpleModule". At this point the module is empty (an empty .modules file has been created as a placeholder for your content).Next, use the reloading mechanism described in the next section to replace this empty module with your content.
Upload Module Content
To load the module on the server, use Upload Module from the module details page of the Admin Console. Note that you do this to populate an empty module as well as to reload it to pick up content you've changed outside the server in the future.
On the > Site > Admin Console > Module Information > Module Details page, find the row for your module.
Click the Upload Module link for the module.
Click Choose File or Browse, and select the .module file to load. Note that here, the name of the module file must be in all lower case. In our example, it is "aloadingdemo.module".
Click Submit.
Now your module can be enabled and used on your server.
Use Your Module
Without starting and stopping the server, your module is now available and the resources can be used whereever you enable the module. If you are using our loading demo module and choose to use it in a folder containing the example study downloadable from this page, you will be able to use a few additional module resources.Navigate to the folder where you want to use the resources it contains and enable it:
Select > Folder > Management.
Click the Folder Type tab.
Check the box for your module name and click Update Folder.
You can now use the contents.
If you used our "aLoadingDemo" module, you can now add a web part of the "1 New Web Part" type.
Edit Your Module Resources
To edit the module-based resources, you must make changes using a development machine (or elsewhere on your filesystem), then regenerate the revised .module file and then use the Upload Module link on the module details page to reload the new content.
When you are working on a Development mode server, you will see the ability to create and load a new module, but this cannot be used to generate an editable version of the module. If you click Edit Module on the module details page, you will see a message indicating it cannot be edited. Instead, you can use your development machine, following the instructions in this topic:
On the Module Details page, on the right side of the row for your module, there are links to:
Edit Module: Learn about editing modules on development servers in the topic: Module Editing Using the Server UI. Note that you cannot edit a module that was loaded "live" as the server does not have access to the source code for it.
Upload Module: Load new information for the module, either by way of a new archive or from the file structure.
Delete Module: Delete the module from the server. Note that deleting a module removes it from the server, and deletes the content in the "build/deploy/modules" subdirectory. It also may cause the webapp to reload.
Troubleshooting
Note that this feature is intended for use on a production server, and will only succeed for modules that do not contain Java code or require compilation.If you see the error message "Archive should not contain java code" and believe that your module does not contain such code, one or more of the following may help you build a module you can load:1. The module.properties file should contain the line:
ModuleClass: org.labkey.api.module.SimpleModule
2. If there is a build.gradle file in your module, it must contain the following:
plugins { id 'org.labkey.build.fileModule' }
3. We recommend that you put your own modules in "<LK_ENLISTMENT>/server/externalModules". (Only LabKey provided modules should be under "<LK_ENLISTMENT>/server/modules" to maintain a clean separation.) Include a build.gradle file in your module's directory, apply the plugin as noted above, and then include it in your settings.gradle file:
Premium Feature — This feature is available with the Professional and Enterprise Editions of LabKey Server. It requires the addition of the moduleEditor module to your distribution. Learn more or contact LabKey.
Edit module-based resources directly within the user interface of a LabKey Server running in development mode. With the moduleEditor module enabled, you can edit many module-based resources without stopping and starting your server.Using this feature on a shared development server lets developers iterate on a module in an environment that may not be convenient to configure on an individual developer's machine, and is well-suited for a shared pre-production testing deployment. It requires that the module's source root is available to the server, in addition to having its built .module deployed. It streamlines the development and maintenance of module-based resources by bypassing the need to edit offline and reload to see each change.
Be sure not to run a production server in "Development mode". Learn more about devmode here.
With Module Editing enabled, two new capabilities are unlocked:
A graphical Module Editor interface makes it easy to find and edit many module-based resources.
Developers can edit some module-based resources in the same way as they would edit them if they had been defined in the user interface. For instance, both SQL Queries and R Reports become editable regardless of where they were defined.
The ModuleEditor module must be deployed on your server.
You must have the necessary permissions to edit the same resources if they were not module-based. For example:
SQL Queries: "Editor" or higher
R Reports: "Platform Developer" or "Trusted Analyst"
2. Create the module's source root by copying it to the server, checking out from source control, or building it manually, as you would on a developer machine. You cannot edit it if you load it through the UI.If you would like an example module to help you get started, you can use this one:
Download and unzip it, then place the "aloadingdemo" folder in your development machine enlistment's "server/modules". If this were not a purely file-based module, you would need to build it as usual.3. Deploy it on the server by running:
./gradlew deployApp
4. Navigate to the folder where you want to be able to use your module-based resources. If you are using our example "aloadingdemo" module which contains a few simple queries and reports, first install the example study you can download from this page:
Check the box for your module, and make sure the ModuleEditor module is also enabled in the folder.
Click Update Folder.
Module Editor
Open the module editor by selecting > Developer Links > Module Editor. You'll see the list of modules in the left hand tree panel, each represented by a folder.Expand the module folders to see the module directory structure and resources inside. You can use the Show Module dropdown to focus on a single module.
Edit Resources
Click any folder or resource to open it for editing in the right hand panel. Your selection is shown in darker type.In the case of resources like the SQL query shown above, you can directly edit the query text in the window and click Save Changes. You could also Move File or Delete File.
Edit Module Structure/Add Content
If you select a folder within the module tree, you have options like creating new subfolders, adding files, and moving or deleting the entire directory.In the above example, we've selected the queries folder. Adding a new subfolder would give us a place to define queries on other schemas (so far there are only queries on the study schema in this module).
Editing Scope and Limitations
You can edit a wide variety of module based resources, including but not limited to:
SQL Queries
Reports
Wikis and other HTML pages
XML resources like views, web parts, query metadata
However, you can only edit content that can be dynamically loaded. Files like the module.properties for a module cannot be editing in this interface.You cannot delete, edit, rename, or upload content directly within the top level of a module.
Edit SQL Queries
SQL queries contained in your module can be edited directly in the module editor above, or also via the query browser, as long as the server has access to the module's source. To be able to edit, you need "Editor" or "Trusted Analyst" permissions. You must also first enable both the ModuleEditor module and the module containing the resources via the > Folder > Management > Folder Type tab.
Select > Go To Module > Query.
Click the schema where your query is defined.
Click the name of the module-defined query. If you are using our example, you can use the study > Master Dashboard query.
In the upper right, you will see a note about where the module is defined, plus options for editing:
Click Edit Source to see the SQL source for your query in the same query editor as when you create a new query. A note will remind you where the query is defined and that changing a module-based query in this folder will also change it in any other folder that uses the same module.From this page you can:
Save & Finish: Save changes and exit the edit interface
Help: See keyboard shortcuts and a link to the SQL Reference.
Changes to will be saved to the module.
Edit Properties
Click Edit Properties to adjust the name, description, availability in child folders, and whether the query is hidden. As with editing of source, be aware that your changes made to the module-resource here will impact all users on the site.
Edit Metadata
Click Edit Metadata to adjust the fields and properties that comprise your query using the field editor. You cannot change the names of fields via this interface, but can set properties, adjust lookups, etc.
Edit R Reports
R reports created in the UI using the R report builder can already be edited. With the ModuleEditor module and the module containing the reports enabled, you can now also edit module-based R reports.Open your R report for editing. If you are using our example module, open the "LymphCD4" report on the Clinical and Assay Data tab. The R Report Builder allows you to edit the report on the Source tab. A message in the UI will alert you that you are editing a module based resource and any changes you make here will be reflected in all other folders where this module is in use.When saved, your report changes will be stored in the original module resources.
Moving Edited Resources to Production
The changes you make to module-based queries and reports via the UI are stored in the root of the module so that you can move the final module to production when ready.Package the module from the development deployment and redeploy on your production server. Learn how in this topic: Deploy Modules to a Production Server
Be sure not to run a production server in "Development mode". Learn more about devmode here.
Use the modules listed below as examples for developing your own modules.To acquire the source code for these modules, enlist in the LabKey Server open source project: Set Up a Development Machine. Many can be downloaded from github or our wiki topics.
Trigger Scripts - Module-based trigger scripts, schemas created by SQL scripts.
LabKey Modules - Descriptions of the modules in the standard distribution of LabKey Server.
Tutorial: File Based Module Resources
This tutorial shows you how to create a file-based module including a variety of reports, queries, and views, and how to surface them in the LabKey Server user interface. Multiple options for module-based resources are included here, including: R reports, SQL queries, SQL query views, HTML views, and web parts.
The Scenario
Suppose that you want to enable a series of R reports, database queries, and HTML views to accompany common study datasets. The end-goal is to deliver these to a client as a unit that can be easily added to their existing LabKey Server installation, and seamlessly made available in all studies at once.Once added, end-users would not be able to modify the queries or reports, ensuring that they keep running as expected. The steps below show how to create these resources using a file-based module.
This tutorial is designed for developers who build LabKey Server from source. Even if you are not a developer and do not build the server from source, you can get a sense of how modules work by installing the module that is the final product of this tutorial, then reading through the steps of the tutorial to see how these resources are surfaced in the user interface.To install the module, download fileBasedDemo.module and deploy it to your server.
To begin this tutorial, we install a target set of data to work with and create the skeleton of our file-based module, the three empty directories:
queries: Holds SQL queries and grid views.
reports: Holds R reports.
views: Holds files to back user interface elements.
Set Up a Dev Machine
Complete the topic below. This will set up your local machine to be able to build LabKey Server from source. While not strictly required for editing file based module resources, this is a typical way a developer might work on custom elements.
On your server, install the example research study in a new folder following the instructions in this topic. The examples in this tutorial use the datasets and columns it contains, though you can make adjustments as required to suit other structures, including those unrelated to study tools.
Within your local dev machine enlistment, go to the <LK_ENLISTMENT>/build/deploy/externalModules/ directory (create it if it doesn't exist), and create the following directory structure and files with these names and extensions:
Note: This module is being created within the built deployment for simplicity. You can start and stop the built server without losing the module you are working on. However, make sure to copy your module folder tree to a location outside the build directory before you run "gradlew cleanBuild" or "gradlew cleanDeploy". If you clean and rebuild ("gradlew deployApp") without first copying your module folder to a more permanent location you will lose your work in progress.
Enable Your Module in a Folder
To use a module, enable it in a folder.
Go to the LabKey Server folder where you want add the module functionality. For this tutorial, go to the study folder where you installed the example study, typically "HIV Study" in the "Tutorials" project.
Select > Folder > Management.
Click the Folder Type tab.
Under the list of Modules click on the check box next to fileBasedDemo to activate it in the current folder.
Click Update Folder.
Now when you add resources to the module, they will be available in this folder.
In a file based module, the queries directory holds SQL queries, and ways to surface those queries in the LabKey Server UI. The following file types are supported:
SQL queries on the database (.SQL files)
Metadata on the above queries (.query.xml files).
Named grid views on pre-existing queries (.qview.xml files)
Trigger scripts attached to a query (.js files) - these scripts are run whenever there an event (insert, update, etc.) on the underlying table.
In this step you will define a custom grid view on the existing "LabResults" table, which is a provisioned table within the example study folder you loaded. Notice that the target schema and query are determined by the directories the file rests inside -- a file named/located at "study/LabResults/SomeView.qview.xml" means "a grid view named SomeView on the LabResults query (dataset) in the study schema".Additionally, if you wish to just create a default view that overrides the system generated one, be sure to just name the file as .qview.xml, so there is no actual root filename. If you use default.qview.xml, this will create another view called "default", but it will not override the existing default grid view.
Create an XML-based Query View
Add two directories (study and LabResults) and a file (DRV Regimen Results.qview.xml), as shown below.
The directory structure tells LabKey Server that the view is in the "study" schema and on the "LabResults" table.
<sorts> may contain multiple sort statements; they will be applied in the order they appear in this section. In this example, we sort descending by the Hemoglobin column. To sort ascending simply omit the descending attribute.
See the Grid View
Once the *.qview.xml file is saved in your module, you can immediately see this custom grid view without rebuilding or restarting.
In your study, click the Clinical and Assay Data tab, then click LabResults.
You can also reach this query via the schema browser: > Go To Module > Query, click study, then LabResults then View Data.
Open the (Grid Views) menu: your custom grid DRV Regimen Results has been added to the list.
In this tutorial step, we add SQL queries to the queries directory. We can also provide associated metadata files to provide additional properties for those queries.
The metadata files are optional, but if provided should have the same name as the .sql file, but with a ".query.xml" extension (e.g., BMI.query.xml) (docs: query.xsd)In this topic, we will create two SQL queries in the study schema, then add metadata to one of them.
Add SQL Queries
Add two .sql files in the queries/study directory, as follows:
SELECT PhysicalExam.ParticipantId, PhysicalExam.Date, PhysicalExam.Weight_kg, Datasets.Demographics.Height*0.01 AS Height_m, Weight_kg/POWER(Datasets.Demographics.Height*0.01, 2) AS BMI FROM PhysicalExam
From the same study folder, open the query browser at > Go To Module > Query.
Click study.
You will see your new queries, listed with the others alphabetically.
Click each query name, then View Data to view the contents.
Add Query Metadata
Using an additional *.query.xml file, you can add metadata including conditional formatting, additional keys, and field formatting. The *.query.xml file should have same root file name as the *.sql file for the query to be modified.Add a new BMI.query.xml file where we can add some metadata for that query:
After saving this metadata file, if you still have a browser viewing the BMI query contents, simply refresh to see the new attributes you just applied: The "Weight (kg)" column header has dropped the parentheses, the excess digits in the calculated BMI column have been truncated, and BMI values over 25 are now highlighted with a yellow background.
In a file-based module, the reports directory holds different kinds of reports and associated configuration files which determine how the reports are surfaced in the user interface.Below we'll make an R report script that is associated with the "MasterDashboard" query (created in the previous step).
In the reports/ directory, create the following subdirectories: schemas/study/MasterDashboard, and a file named "HgbHistogram.r", as shown below:
In your study folder, open the query browser via > Go to Module > Query.
Click study, then MasterDashboard.
Click View Data to run the query and view the results.
View the new R report by selecting (Charts/Reports) > HgbHistogram.
Add Another Report
You can also add R reports on the "built-in" queries, which include provisioned datasets. To do so, name the directory for the table instead of the query. For example:
In a module, the views directory holds user interface elements, like HTML pages and associated web parts that help your users view your resources.In this topic, we will provide an HTML view making it easier for our users to see the custom queries and reports created in the tutorial earlier. Otherwise, users would need to access the query schema to find them. You could provide this in a wiki page created on the server, but our goal is to provide everything in the module itself. Here we will create an HTML view and an associated web part.
Under the views/ directory, create a new file named dashboardDemo.html, and enter the following HTML:
<p> <a id="dash-report-link" href="<%=contextPath%><%=containerPath%>/query-executeQuery.view?schemaName=study&query.queryName=MasterDashboard"> Dashboard of Consolidated Data</a> </p>
While queries and report names may contain spaces, note that .html view files must not contain spaces. The view servlet expects that action names do not contain spaces.
View Page via URL
You can directly access module resources using the URL. The pattern to directly access module based resources is as follows:
The action is the combination of your resource name and an action suffix, typically ".view" or ".post". For example, if you installed the example study on a localhost in a "Tutorials/HIV Study" folder, you can directly access the Dashboard Demo HTML page here:
In the HTML you pasted, notice the use of the <%=contextPath%> and <%=containerPath%> tokens in the URL's href attribute. Since the href in this case needs to be valid in various locations, we don't want to use a fixed path. Instead, these tokens will be replaced with the server's context path and the current container path respectively.Learn more about token replacement/expansion in this topic:
Web resources such as images, javascript, and other html files can be placed in the /web directory in the root of the module. To reference an image from one of the views pages, use a url including the context path such as:
<img src="<%=contextPath%>/my-image.png" />
Define a View Wrapper
This file has the same base-name as the HTML file, "dashboardDemo", but with an extension of ".view.xml". In this case, the file should be called dashboard.view.xml, and it should contain the following:
To allow this view to be added as a web part, create our final file, the web part definition. Create a file in the views/ directory called dashboardDemo.webpart.xml and enter the following content, referring to the root name of the file:
After creating these files, you refresh the portal page in your folder, enter > Page Admin Mode and see the Data Dashboard web part as an option to add on the left side of the page. Add it to the page, and it will display the contents of the dashboardDemo.html view. In this case, a direct link to your module-defined query.The user can access the Histogram report via the (Charts/Reports) menu, or you can make it even easier with another direct link as described in the next section.
Add More Content to the Web Part
Now that you have established the user interface, you can directly modify the HTML file to change what the user can use on existing web parts.Reopen dashboardDemo.html for editing and add this below the existing content:
On the page where you have your Data Dashboard web part, simply refresh the page, and you'll see a new Hemoglobin Histogram link directly to your R report.
Set Required Permissions
You might also want to require specific permissions to see this view. That is easily added to the dashboardDemo.view.xml file like this:
In this case, once you have added this permission element, only admins will be able to see the dashboard.Possible permissionClass names include, but are not limited to:
ReadPermission
InsertPermission
UpdatePermission
AdminPermission
If there is no .view.xml file at all, or no "<requiresPermissions>" element is included, it is assumed that both login and read permission are required. In order to disable this automatic default permissioning, use "<requiresNoPermission>".
The XSD for this meta-data file is view.xsd in the schemas/ directory of the project. The LabKey XML Schema Reference provides an easy way to navigate the documentation for view.xsd.
Congratulations
You have completed our tutorial for learning to create and use file-based modules to present customized content to your users.
To use a JavaScript library in your module, do the following:
Acquire the library .js file you want to use.
In your module resources directory, create a subdirectory named "web".
Inside "web", create a subdirectory with the same name as your module. For example, if your module is named 'helloworld', create the following directory structure:
helloworld
└───resources
└───web
└───helloworld
Copy the library .js file into your directory structure. For example, if you wish to use a JQuery library, place the library file as shown below:
Note: if you declare dependencies explicitly in the .view.xml file, you don't need to use LABKEY.requiresScript on the HTML page.
Remote Dependencies
In some cases, you can declare your dependency using an URL that points directly to the remote library, instead of copying the library file and distributing it with your module:
Module-based assays allow a developer to create a new assay type with a custom schema and custom views without becoming a Java developer. A module-based assay type consists of an assay config file, a set of domain descriptions, and view html files. The assay is added to a module by placing it in an assay directory at the top-level of the module. When the module is enabled in a folder, assay designs can be created based on the type defined in the module. For information on the applicable JS API, see: LABKEY.Experiment#saveBatch.
The assay consists of an assay config file, a set of domain descriptions, and view html files. The assay is added to a module by placing it in an assay directory at the top-level of the module. The assay has the following file structure:
<module-name>/
assay/
ASSAY_NAME/
config.xml example
domains/ - example
batch.xml
run.xml
result.xml
views/ - example
begin.html
upload.html
batches.html
batch.html
runs.html
run.html
results.html
result.html
queries/ - example
Batches.query.xml
Run.query.xml
Data.query.xml
CUSTOM_ASSAY_QUERY.query.xml
CUSTOM_ASSAY_QUERY.sql
CUSTOM_ASSAY_QUERY/
CUSTOM_VIEW.qview.xml
scripts/
script1.R
script2.pl
The only required part of the assay is the <assay-name> directory. The config.xml, domain files, and view files are all optional. The CUSTOM_ASSAY_QUERY.sql will shows up in the schema for all assay designs of this provider type.This diagram shows the relationship between the pages. The details link will only appear if the corresponding details html view is available.
How to Specify an Assay "Begin" Page
Module-based assays can be designed to jump to a "begin" page instead of a "runs" page. If an assay has a begin.html in the assay/<name>/views/ directory, users are directed to this page instead of the runs page when they click on the name of the assay in the assay list.
A domain is a collection of fields that describe a set of data. Each type of data (e.g., Assays, Lists, Datasets, etc.) provides specialized handling for the domains it defines. Assays define multiple domains, while Lists and Datasets define only one domain each. This topic will help you understand how to build a domain for your module-based custom assay type.There are three built-in domains for assay types: "batch", "run", and "result". An assay module can define a custom domain by adding a schema definition in the domains/ directory. For example:
assay/<assay-name>/domains/<domain-name>.xml
The name of the assay is taken from the <assay-name> directory. The contents of <domain-name>.xml file contains the domain definition and conforms to the <domain> element from assayProvider.xsd, which is in turn a DomainDescriptorType from the expTypes.xsd XML schema. This following result domain replaces the built-in result domain for assays:
result.xml
<ap:domain xmlns:exp="http://cpas.fhcrc.org/exp/xml" xmlns:ap="http://labkey.org/study/assay/xml"> <exp:Description>This is my data domain.</exp:Description> <exp:PropertyDescriptor> <exp:Name>SampleId</exp:Name> <exp:Description>The Sample Id</exp:Description> <exp:Required>true</exp:Required> <exp:RangeURI>http://www.w3.org/2001/XMLSchema#string</exp:RangeURI> <exp:Label>Sample Id</exp:Label> </exp:PropertyDescriptor> <exp:PropertyDescriptor> <exp:Name>TimePoint</exp:Name> <exp:Required>true</exp:Required> <exp:RangeURI>http://www.w3.org/2001/XMLSchema#dateTime</exp:RangeURI> </exp:PropertyDescriptor> <exp:PropertyDescriptor> <exp:Name>DoubleData</exp:Name> <exp:RangeURI>http://www.w3.org/2001/XMLSchema#double</exp:RangeURI> </exp:PropertyDescriptor> </ap:domain>
To deploy the module, the assay directory is zipped up as a <module-name>.module file and copied to the LabKey server's modules directory.When you create a new assay design for that assay type, it will use the fields defined in the XML domain as a template for the corresponding domain. Changes to the domains in the XML files will not affect existing assay designs that have already been created.
This topic describes how to customize a module-based assay, adding a [details] link for each row of result data. These [details] links take you to a custom details view for that row. Similarly, you can add custom views for runs and batches.
You can add new views to the module-based assay by adding html files in the views/ directory, for example:
assay/<assay-name>/views/<view-name>.html
The overall page template will include JavaScript objects as context so that they're available within the view, avoiding an extra client API request to fetch it from the server. For example, the result.html page can access the assay definition and result data as LABKEY.page.assay and LABKEY.page.result respectively. Here is an example custom details view named result.html:
Note on line 28 the details view is accessing the result data from LABKEY.page.result. See Example Assay JavaScript Objects for a description of the LABKEY.page.assay and LABKEY.page.result objects.
Add a Custom View for a Run
For a custom view for a run, you use the same process as for the custom details page for the row data shown above, except the view file name is run.html and the run data will be available as the LABKEY.page.run variable. See Example Assay JavaScript Objects for a description of the LABKEY.page.run object.
Add a Custom View for a Batch
For a custom view for a batch, you use the same process as above, except the view file name is batch.html and the run data will be available as the LABKEY.page.batch variable. See Example Assay JavaScript Objects for a description of the LABKEY.page.batch object.
Module based custom views are loaded based on their association with the target query name. Loading by table title in past versions caused significant performance issues and is no longer supported.
Loading Custom Views
To attach a custom view to a table, bind it via the query name. For instance, if you have a query in the elispot module called QueryName, which includes the table name definition as TableTitle, and your custom view is called MyView, you would place the xml file here:
The JavaScript objects for assays described in this topic are automatically injected into the rendered page (example page: result.html), to save developers from needing to make a separate JavaScript client API request via AJAX to separately fetch them from the server.
The batch object is available as LABKEY.page.batch on the upload.html and batch.html pages. The JavaScript object is an instance of Run Group and is shaped like:
LABKEY.page.batch = new LABKEY.Exp.RunGroup({ "id": 8, "createdBy": <user name>, "created": "8 Apr 2009 12:53:46 -0700", "modifiedBy": <user name>, "name": <name of the batch object>, "runs": [ // array of LABKEY.Exp.Run objects in the batch. See next section. ], // map of batch properties "properties": { "ParticipantVisitResolver": null, "TargetStudy": null }, "comment": null, "modified": "8 Apr 2009 12:53:46 -0700", "lsid": "urn:lsid:labkey.com:Experiment.Folder-5:2009-04-08+batch+2" });
LABKEY.page.run:
The run detail object is available as LABKEY.page.run on the run.html pages. The JavaScript object is an instance of LABKEY.Exp.Run and is shaped like:
LABKEY.page.run = new LABKEY.Exp.Run({ "id": 4, // array of LABKEY.Exp.Data objects added to the run "dataInputs": [{ "id": 4, "created": "8 Apr 2009 12:53:46 -0700", "name": "run01.tsv", "dataFileURL": "file:/C:/Temp/assaydata/run01.tsv", "modified": null, "lsid": <filled in by the server> }], // array of objects, one for each row in the result domain "dataRows": [ { "DoubleData": 3.2, "SampleId": "Monkey 1", "TimePoint": "1 Nov 2008 11:22:33 -0700" }, { "DoubleData": 2.2, "SampleId": "Monkey 2", "TimePoint": "1 Nov 2008 14:00:01 -0700" }, { "DoubleData": 1.2, "SampleId": "Monkey 3", "TimePoint": "1 Nov 2008 14:00:01 -0700" }, { "DoubleData": 1.2, "SampleId": "Monkey 4", "TimePoint": "1 Nov 2008 00:00:00 -0700" } ], "createdBy": <user name>, "created": "8 Apr 2009 12:53:47 -0700", "modifiedBy": <user name>, "name": <name of the run>, // map of run properties "properties": {"DoubleRun": null}, "comment": null, "modified": "8 Apr 2009 12:53:47 -0700", "lsid": "urn:lsid:labkey.com:SimpleRun.Folder-5:cf1fea1d-06a3-102c-8680-2dc22b3b435f" });
LABKEY.page.result:
The result detail object is available as LABKEY.page.result on the result.html page. The JavaScript object is a map for a single row and is shaped like:
You can associate query metadata with an individual assay design, or all assay designs that are based on the same type of assay (e.g., "NAb" or "Viability"). This topic describes how to associate metadata with either scope.
Assay table names are based upon the name of the assay design. For example, consider an assay design named "MyViability" that is based on the "Viability" assay type. This design would be associated with three tables in the schema explorer: "MyViability Batches", "MyViability Runs", and "MyViability Data."
Associate Metadata with a Single Assay Design
To attach query metadata to the "MyViability Data" table, you would normally create a /queries/assay/MyViability Data.query.xml metadata file. This would work well for the "MyViability Data" table itself. However, this method would not allow you to re-use this metadata file for a new assay design that is also based on the same assay type ("Viability" in this case).
Associate Metadata with All Assay Designs Based on a Particular Assay Type
To enable re-use of the metadata, you need to create a query metadata file whose name is based upon the assay type and table name. To continue our example, you would create a query metadata file called /assay/Viability/queries/Data.query.xml to attach query metadata to all data tables based on the Viability-type assay, including your "MyViability" instance.As with other query metadata in module files, the module must be enabled on the > Folder > Management > Folder Type tab. Learn more in this topic: Project and Folder Settings.
You can enable file-based assays to customize their own Experiment.saveBatch behavior by writing Java code that implements the AssaySaveHandler interface. This allows you to customize saving your batch without having to convert your existing file-based assay UI code, queries, views, etc. into a Java-based assay.
The AssaySaveHandler interface enables file-based assays to extend the functionality of the SaveAssayBatch action with Java code. A file-based assay can provide an implementation of this interface by creating a Java-based module and then putting the class under the module's src directory. This class can then be referenced by name in the <saveHandler/> element in the assay's config file. For example, an entry might look like:
Create the skeleton framework for a Java module. This consists of a controller class, manager, etc. Find details about autogenerating the boiler plate Java code in this topic: Tutorial: Hello World Java Module.
Add an assay directory underneath the Java src directory that corresponds to the file-based assay you want to extend. For example:
myModule/src/org.labkey.mymodule/assay/tracking
Implement the AssaySaveHandler interface. You can choose to either implement the interface from scratch or extend default behavior by having your class inherit from the DefaultAssaySaveHandler class. If you want complete control over the JSON format of the experiment data you want to save, you may choose to implement the AssaySaveHandler interface entirely. If you want to follow the pre-defined LABKEY experiment JSON format, then you can inherit from the DefaultAssaySaveHandler class and only override the specific piece you want to customize. For example, you may want custom code to run when a specific property is saved. See more details below.
Reference your class in the assay's config.xml file. For example, notice the <ap:saveHandler/> entry below. If a non-fully-qualified name is used (as below) then LabKey Server will attempt to find this class under org.labkey.[module name].assay.[assay name].[save handler name].
<ap:provider xmlns:ap="http://labkey.org/study/assay/xml"> <ap:name>Flask Tracking</ap:name> <ap:description> Enables entry of a set of initial samples and then tracks their progress over time via a series of daily measurements. </ap:description> <ap:saveHandler>TrackingSaveHandler</ap:saveHandler> <ap:fieldKeys> <ap:participantId>Run/PatientId</ap:participantId> <ap:date>MeasurementDate</ap:date> </ap:fieldKeys> </ap:provider>
The interface methods are invoked when the user imports data into the assay or otherwise calls the SaveAssayBatch action. This is usually invoked by the Experiment.saveBatch JavaScript API. On the server, the file-based assay provider will look for an AssaySaveHandler specified in the config.xml and invoke its functions. If no AssaySaveHandler is specified then the DefaultAssaySaveHandler implementation is used.
SaveAssayBatch Details
The SaveAssayBatch function creates a new instance of the SaveHandler for each request. SaveAssayBatch will dispatch to the methods of this interface according to the format of the JSON Experiment Batch (or run group) sent to it by the client. If a client chooses to implement this interface directly then the order of method calls will be:
beforeSave
handleBatch
afterSave
A client can also inherit from DefaultAssaySaveHandler class to get a default implementation. In this case, the default handler does a deep walk through all the runs in a batch, inputs, outputs, materials, and properties. The sequence of calls for DefaultAssaySaveHandler are:
beforeSave
handleBatch
handleProperties (for the batch)
handleRun (for each run)
handleProperties (for the run)
handleProtocolApplications
handleData (for each data output)
handleProperties (for the data)
handleMaterial (for each input material)
handleProperties (for the material)
handleMaterial (for each output material)
handleProperties (for the material)
afterSave
Because LabKey Server creates a new instance of the specified SaveHandler for each request, your implementation can preserve instance state across interface method calls within a single request but not across requests.
SQL Scripts for Module-Based Assays
How do you add supporting tables or lists to your assay type? For example, suppose you want to add a table of Reagents, which your assay domain refers to via a lookup/foreign key?Some options:
Manually import a list archive into the target folder. Effective, but manual and repetitive.
Add the table/lists via SQL scripts included in the module. This option allows you to automatically include your list in any folder where your assay is enabled.
Add Supporting Table
To insert data: use SQL DML scripts or create an initialize.html view that populates the table using LABKEY.Query.insertRows().To add the supporting table using SQL scripts, add a schemas directory, as a sibling to the assay directory, as shown below.
If your assay supports only one database, i.e. "pgsql", you can include a script only for that database, and configure your module properties accordingly. Learn more under "SupportedDatabases" in module.properties Reference.LabKey Server does not currently support adding assay types or lists via SQL scripts, but you can create a new schema to hold the table, for example, the following script creates a new schema called "myreagents" (on PostgreSQL):
DROP SCHEMA IF EXISTS myreagents CASCADE;
CREATE SCHEMA myreagents;
CREATE TABLE myreagents.Reagents ( RowId SERIAL NOT NULL, ReagentName VARCHAR(30) NOT NULL
);
ALTER TABLE ONLY myreagents.Reagents ADD CONSTRAINT Reagents_pkey PRIMARY KEY (RowId);
INSERT INTO myreagents.Reagents (ReagentName) VALUES ('Acetic Acid'); INSERT INTO myreagents.Reagents (ReagentName) VALUES ('Baeyers Reagent'); INSERT INTO myreagents.Reagents (ReagentName) VALUES ('Carbon Disulfide');
Update the assay domain, adding a lookup/foreign key property to the Reagents table:
If you'd like to allow admins to add/remove fields from the table, you can add an LSID column to your table and make it a foreign key to the exp.Object.ObjectUri column in the schema.xml file. This will allow you to define a domain for the table much like a list. The domain is per-folder so different containers may have different sets of fields.For example, see reagent/resources/schemas/reagent.xml. It wires up the LSID lookup to the exp.Object.ObjectUri column
Transform scripts are attached to assay designs, run before the assay data is imported, and can reshape the data file to match the expected import format. Several scripts can run sequentially to perform different transformations. The extension of the script file identifies the scripting engine that will be used to run the validation script. For example, a script named test.pl will be run with the Perl scripting engine.
Transform scripts (which are always attached to assay designs) are different from trigger scripts, which are attached to other table types (datasets, lists, sample types, etc.).
Scenarios
A wide range of scenarios can be addressed using transform scripts. For example:
Input files might not be "parsable" as is. Instrument-generated files often contain header lines before the main data table, denoted by a leading #, !, or other symbol. These lines may contain useful metadata about the protocol, reagents, or samples tested which should either be incorporated into the data import or skipped over to find the main data to import.
File or data formats in the file might not be optimized for efficient storage and retrieval. Display formatting, special characters, etc. might be unnecessary for import. Transformation scripts can clean, validate, and reformat imported data.
During import, display values from a lookup column may need to be mapped to foreign key values for storage.
You may need to fill in additional quality control values with imported assay data, or calculate contents of a new column from columns in the imported data.
Inspect and change the data or populate empty columns in the data. Modify run- and batch-level properties. If validation only needs to be done for particular single field values, the simpler mechanism is to use a validator within the field properties for the column.
Scripting Prerequisites
Any scripting language that can be invoked via the command line and has the ability to read/write files is supported for transformation scripts, including:
Before you can run scripts, you must configure the necessary scripting engine on your server. If you are missing the necessary engine, or the desired engine does not have the script file extension you are using, you'll get an error message similar to one of these:
Script engine for the extension '.pl' has not been registered.
A script engine implementation was not found for the specified QC script (my_transformation_script.py). Check configurations in the Admin Console.
Permissions
In order to upload transform scripts and attach them to an assay design, the user must have either the Platform Developer or Site Administrator role. Once an authorized user has added a script, it will be run any time data is imported using that design.Users who can edit assay designs but are not Platform Developers or Site Administrators will be able to edit other aspects of the design, but will not see the transformation script options.
How Transformation Scripts Work
Transformation and validation scripts invoked during data import follow this Script Execution Sequence:1. A user imports assay result data and supplies run and batch properties.2. The server uses that input to create:
(1) A "runProperties.tsv" file.
This file contains assay-specific properties, run and batch level fields, and generated paths to various resources, all of which can be used by your transformation script.
(2) A "runDataFile" in TSV format, if possible. The path to the intended location is included in the runProperties.tsv file regardless of whether the server can generate the "runDataFile" itself. See below.
(2a) If the first 'non-comment' line of the imported result data contains fields that match the assay design, the server can infer/generate a TSV format "runDataFile" based on the result data provided by the user (which may be in another file format like Excel or CSV). This preprocessed data TSV can then be transformed.
(2b) If the server cannot infer this file, either because the "relevant" part is not at the top, column names are not recognizable to the assay mechanism, or generally isn't tabular/'rectangular', you cannot use the generated "runDataFile", but will instead be able to use the originally uploaded file in its original format (as "runDataUploadedFile").
A pair of examples of reading in these two types of input file is included in this topic.
3. The server invokes the transform script, which can use the ${runInfo} substitution token to locate the "runProperties.tsv" file created in step 2. The script can then extract information and paths to input and output files from that file.
Both the "runDataUploadedFile" (the raw data file you uploaded) and the "runDataFile" (the location for the processed/TSV version of that file that may or may not have been generated in step 2) are included in the "runProperties.tsv" file. Which you use as input depends on your assay type and transformation needs.
The "runDataFile" property consists of three columns; the third of which is the path to the output TSV file you will write the data to.
4. After script completion, the server checks whether any errors have been written by the transform script and whether any data has been transformed.5. If transformed data is available in the specified output location, the server uses it for subsequent steps; otherwise, the original data is used.6. If multiple transform scripts are specified, the server invokes the other scripts in the order in which they are defined, passing sequentially transformed output as input to the next script.7. Field-level validator and quality-control checks, including range and regular expression validation that are included in the assay definition are performed on the 'post-transformation' data.8. If no errors have occurred, the run is loaded into the database.
Use Transformation Scripts
Each assay design can be associated with one or more validation or transform scripts which are run in the order listed in the assay design.For a walkthrough of configuring and using using a transform script that has already been developed for your assay type, follow this topic:
To use a transform script in an assay design, edit the design and click Add Script next to the Transform Scripts field. Note that you must have Platform Developer or Site Administrator to see or use this option.
For each script you add, use the checkboxes to control when that script is run:
Run on Import: When any authorized user imports (or re-imports) data using this assay design, the script will be executed.
Run on Edit: When any authorized user edits data using this assay design, the script will be run.
Save your changes.
There are two other useful Import Settings available in the Assay designer.
Save Script Data for Debugging tells the framework to not delete the intermediate files such as the runProperties file after a successful run. This option is important during script development. It can be turned off to avoid cluttering the file space under the TransformAndValidationFiles directory that the framework automatically creates under the script file directory.
Import In Background tells the framework to create a pipeline job as part of the import process, rather than tying up the browser session. It is useful for importing large data sets.
A few notes on usage:
Transform scripts are triggered both when a user imports via the server graphical user interface and when the client API initiates the import (for example via saveBatch).
Client API calls are not supported in the body transform scripts, only server-side code is supported.
Columns populated by transform scripts must already exist in the assay definition.
Executed scripts show up in the experimental graph, providing a record that transformations and/or quality control scripts were run.
Transform scripts are run before field-level validators.
The script is invoked once per run upload.
Multiple scripts are invoked in the order they are listed in the assay design.
Note that non-programmatic quality control remains available -- assay designs can be configured to perform basic checks for data types, required values, regular expressions, and ranges. Learn more in these topics: Field Editor and Dataset QC States: Admin Guide.
The general purpose assay tutorial includes another example use of a transform script in Assay Transform Script.
Manage Script Files
When you add a transformation script using the assay designer, the script will be uploaded to a @scripts subdirectory of the file root, parallel to where other @files are stored. This separate location helps protect scripts from being modified or removed by unauthorized users, as only Platform Developers and Site Administrators will be able to access them.Remove scripts from the design by selecting Remove path from the menu. Note that this does not remove the file itself, just removes the path from the assay design. You can also use Copy path to obtain the path for this script in order to apply it to another assay design.To manage the actual script files, click Manage Script Files to open the @scripts location.Here you can select and (Delete) the script files themselves.
Customize a File Browser
You can customize a Files web part to show the @scripts location.
Choose Customize from the menu of the new web part.
Select the @scripts root.
You may also want to customize the title of the web part.
Note: If users without access to the script files will be able to reach this folder, you will also want to customize the permissions settings of this web part. This is best accomplished by creating a dummy container and granting only Site Administrators and Platform Developers access, then using that container to set the permissions for the Transform Scripts web part.
Provide the Path to the Script File
When you upload a transformation script to the assay designer, it is placed in the @scripts subdirectory of the local file root. The path is determined for you and displayed in the assay designer. This location is only visible to Site Administrators and users with the Platform Developer role, making it a secure place to locate script files.If for some reason you have scripts located elsewhere on your system, or when you are creating a new design using the same transform script(s), you can specify the absolute path to the script instead of uploading it.Use > Copy path from an existing assay design's transform script section, or find the absolute path of a script elsewhere in the File Repository.
If there is a customized Files web part showing the contents of the @scripts location, as shown here, click the title of the web part to open the file browser.
Select the transform script.
The absolute path is shown at the bottom of the panel.
In the file path, LabKey Server accepts either backslashes (the default Windows format) or forward slashes.Example path to script:
When working on your own developer workstation, you can put the script file wherever you like, but using the assay designer interface to place it in the @scripts location will not only be more secure, but will also make it easier to deploy to a production server. These options also make iterative development against a remote server easier, since you can use a Web-DAV enabled file editor to directly edit the same script file that the server is calling.Within the script, you can use the built-in substitution token "${srcDirectory}" which is automatically the directory where the script file is located.
Access and Use the Run Properties File
The primary mechanism for communication between the LabKey Assay framework and the Transform script is the Run Properties file. The ${runInfo} substitution token tells the script code where to find this file. The script file should contain a line like
The run properties file contains three categories of properties:1. Batch and run properties as defined by the user when creating an assay instance. These properties are of the format:
<property name> <property value> <java data type>for example,
gDarkStdDev 1.98223 java.lang.Double
An example Run Properties file to examine: runProperties.tsvWhen the transform script is called these properties will contain any values that the user has typed into the "Batch Properties" and "Run Properties" sections of the import form. The transform script can assign or modify these properties based on calculations or by reading them from the raw data file from the instrument. The script must then write the modified properties file to the location specified by the transformedRunPropertiesFile property.2. Context properties of the assay such as assayName, runComments, and containerPath. These are recorded in the same format as the user-defined batch and run properties, but they cannot be overwritten by the script.3. Paths to input and output files. These are absolute paths that the script reads from or writes to. They are in a <property name> <property value> format without property types. The paths currently used are:
a. runDataUploadedFile: The assay result file selected and imported to the server by the user. This can be an Excel file (XLS, XLSX), a tab-separated text file (TSV), or a comma-separated text file (CSV).
b. runDataFile: The file produced after the assay framework converts the user imported file to TSV format. The path will point to a subfolder below the script file directory, with a path value similar to <property value> <java property type>. The AssayId_22\42 part of the directory path serves to separate the temporary files from multiple executions by multiple scripts in the same folder.