To assist in debugging errors, an administrator can force the LabKey Server process to dump the list of threads and memory to disk. This information can then be sent to LabKey for review.

A thread dump is useful for diagnosing issues where LabKey Server is hung or some requests spin forever in the web browser.

A memory/heap dump is useful for diagnosing issues where LabKey Server is running out of memory, or in some cases where the process is sluggish and consuming a lot of CPU performing garbage collection.

Definitions:

  • LABKEY_HOME: Installation location of the LabKey Server
    • If you performed a Manual Installation and followed these instructions, then this is most likely
      • /usr/local/labkey
    • If you used our Windows installer, then this is most likely one of the following:
      • C:\Program Files (x86)\LabKey Server\
      • C:\Program Files\LabKey Server\
  • CATALINA_HOME: Installation location of the Apache Tomcat Web Server
    • If you performed a Manual Installation and followed these instructions, then this is most likely
      • /usr/local/tomcat
    • If you used our Windows installer, then this is most likely one of the following:
      • C:\Program Files (x86)\LabKey Server\apache-tomcat-*.*.*
      • C:\Program Files\LabKey Server\apache-tomcat-*.*.*

Thread Dump

To dump the list of running threads, you can use either the UI or the command line.

Using the UI

  • Go to (Admin) > Site > Admin Console.
  • Select the Admin Console Links tab.
  • Under Diagnostics, click Running Threads
  • This will show the state of all active threads in the browser, as well as writing the thread dump to the server log file.

Manually via the Command Line

The server monitors the timestamp of a threadDumpRequest file and when it notices a change, it initiates the thread dump process. The content of the file does not matter.

Unix. On a Unix server (Linux, MacOSX, Solaris), use the command line to execute the following commands:

  • Change to the `LABKEY_HOME` directory. For example, if your 'LABKEY_HOME' directory is located at `/usr/local/labkey`, then the command will be
cd /usr/local/labkey
  • Force the server to dump its memory:
touch threadDumpRequest

Windows. On a Windows Server, do the following:

  • Open a Command Prompt .
  • Change to the `LABKEY_HOME` directory. For example, if your 'LABKEY_HOME' directory is located at `C:\Program Files\LabKey Server\`, then the command will be:
cd "C:\Program Files\LabKey Server"
  • Force the server to dump its memory. The command will open the heapDumpRequest file in the notepad program.
notepad threadDumpRequest
  • Place the cursor at the top of the file and hit the "Enter" key twice.
    • Save the file.
    • Close notepad.
Location of the file containing the thread dump

The list of threads is dumped the `labkey.log` file, which is located in the `CATALINA_HOME\logs`

Memory Dump

To request LabKey Server dump its memory, you can use either the UI or the command line. By default, it will write the file to the LABKEY_HOME directory. Java can also be configured to dump the heap if the virtual machine runs out of memory.

Heap dumps can be large: several gigabytes in size is typical. You can tell the server to write to a different location by using a JVM startup argument,

-XX:HeapDumpPath=/path/to/desired/target

Using the UI

  • Go to (Admin) > Site > Admin Console.
  • Select the Admin Console Links tab.
  • Under Diagnostics, click Dump Heap.
  • You will see a full path to where the heap was dumped.

Manually via the Command Line

The server monitors the timestamp of a heapDumpRequest file and when it notices a change, it initiates the heap dump process. The content of the file does not matter.

Unix. On a Unix server (Linux, MacOSX, Solaris), execute the following commands on the command line:

  • Navigate to the LABKEY_HOME directory.
  • Force the server to dump its memory
touch heapDumpRequest

Windows. On a Windows Server, do the following:

  • Open a Command Prompt
  • Navigate to the LABKEY_HOME directory.
  • Force the server to dump its memory. This command will open the heapDumpRequest file in the notepad program.
notepad heapDumpRequest
  • Place the cursor at the top of the file and hit the Enter key twice.
    • Save the file
    • Close notepad
Location of the file containing the memory dump

The file will be located in the LABKEY_HOME directory. The file will have the '.hprof' extension.

Automatically Capturing Heap Dump when Server Runs Out of Memory

The Java virtual machine supports an option that will automatically trigger a heap dump when the server runs out of memory. This is accomplished via this JVM startup parameter:

-XX:+HeapDumpOnOutOfMemoryError

For more information, see Oracle's documentation.

Debugging Postgres Errors

LabKey server is unable to log errors thrown by Postgres, so in the case of diagnosing some installation and startup errors, it may be helpful to to view the event log.

On Windows:
  • Launch eventvwr.msc
  • Navigate to Windows Logs > Application.
  • Search for errors there corresponding to the installation failures, which may assist LabKey support in diagnosing the problem.
  • If you can't find relevant messages, you may be able to trigger the error to occur again by running net start LabKey_pgsql-9.2 from the command line.

In addition, you may find assistance debugging PostgreSQL on their troubleshooting installation page. Included there are instructions to 'Collect the installer log file'.

Collect SQL Server Information

To get a full picture of some problems, it's also useful to have information about running queries and locks. For this, you want both a thread dump (described above) and information about the state of database connections. The latter needs to be obtained directly from SQL Server.

  • Launch SQL Server Management Console or a similar tool
  • Open a connection to the LabKey Server database, often named 'labkey'.
  • Run and capture the output from the following queries/stored procedures:
    • sp_lock
    • sp_who2
    • SELECT t1.resource_type, t1.resource_database_id, t1.resource_associated_entity_id, t1.request_mode, t1.request_session_id, t2.blocking_session_id FROM sys.dm_tran_locks as t1 INNER JOIN sys.dm_os_waiting_tasks as t2 ON t1.lock_owner_address = t2.resource_address;

Discussion

Was this content helpful?

Log in or register an account to provide feedback


previousnext
 
expand all collapse all