To assist in debugging errors, an administrator can force the LabKey Server process to dump the list of threads and memory to disk. This information can then be reviewed to determine the source of the error.
A
thread dump is useful for diagnosing issues where LabKey Server is hung or some requests spin forever in the web browser.
A
memory/heap dump is useful for diagnosing issues where LabKey Server is running out of memory, or in some cases where the process is sluggish and consuming a lot of CPU performing garbage collection.
Definitions
- <LABKEY_HOME>: Installation location of LabKey Server, for example:
- Linux: /labkey/labkey
- Windows: C:\labkey\labkey\
Thread Dump
Examining the state of all running threads is useful when troubleshooting issues like high CPU utilization, or requests that are taking longer than expected to process. To dump the state of all of running threads, you can use either the UI or the command line.
Using the UI
- Go to (Admin) > Site > Admin Console.
- Under Diagnostics, click Running Threads
- This will show the state of all active threads in the browser, as well as writing the thread dump to the server log file.
Manually via the Command Line
The server monitors the timestamp of a threadDumpRequest file and when it notices a change, it initiates the thread dump process. The content of the file does not matter.
Unix. On Linux and OSX, use the command line to execute the following commands:
- Navigate to the <LABKEY_HOME> directory.
- Force the server to dump its memory by running:
Windows. On a Windows Server, do the following:
- Open a Command Prompt.
- Navigate to the <LABKEY_HOME> directory.
- Force the server to dump its memory. This command will open the threadDumpRequest file in Notepad.
notepad threadDumpRequest
- Edit this file to make any trivial change (example: add two newlines at the top of the file).
- Save the file and close Notepad.
Location of the File Containing the Thread Dump
The list of threads is dumped into the <LABKEY_HOME>/logs/labkey.log file.
Memory Dump (Heap Dump)
To request LabKey Server dump its memory, you can use either the UI or the command line. By default, it will write the file to the <LABKEY_HOME> directory. Java can also be configured to automatically dump the heap if the virtual machine runs out of memory and throws an OutOfMemoryError.
The heap dump can be useful to determine what is consuming memory in cases where the LabKey Server process is running out of memory. Note that heap dumps are large since they capture the contents of the full Java memory usage. Several gigabytes in size is typical. You can tell the server to write to a different location by using a JVM startup argument:
-XX:HeapDumpPath=/path/to/desired/target
Using the UI
- Go to (Admin) > Site > Admin Console.
- Under Diagnostics, click Dump Heap.
- You will see a full path to where the heap was dumped.
Manually via the Command Line
The server monitors the timestamp of a heapDumpRequest file and when it notices a change, it initiates the heap dump process. The content of the file does not matter.
Unix. On Linux and OSX, execute the following commands on the command line:
- Navigate to the <LABKEY_HOME> directory.
- Force the server to dump its memory by running:
Windows. On a Windows Server, do the following:
- Open a Command Prompt
- Navigate to the <LABKEY_HOME> directory.
- Force the server to dump its memory. This command will open the heapDumpRequest file in the notepad:
- Edit this file to make any trivial change (example: add a newline at the top of the file).
- Save the file and close Notepad.
Location of the File Containing the Memory Dump
The file will be located in the <LABKEY_HOME> directory. The file will have the ".hprof" extension.
Automatically Capturing Heap Dump when Server Runs Out of Memory
The Java virtual machine supports an option that will automatically trigger a heap dump when the server runs out of memory. This is accomplished via this JVM startup parameter:
-XX:+HeapDumpOnOutOfMemoryError
For more information, see
Oracle's documentation.
Debugging PostgreSQL Errors
LabKey Server is unable to log errors thrown by PostgreSQL, so in the case of diagnosing some installation and startup errors, it may be helpful to view the event log.
On Windows:
- Launch eventvwr.msc
- Navigate to Windows Logs > Application.
- Search for errors there corresponding to the installation failures.
- If you can't find relevant messages, you may be able to trigger the error to occur again by running net start LabKey_pgsql-9.2 (with the name of the Postgres service on your server) from the command line.
In addition, you may find assistance debugging PostgreSQL on their
troubleshooting installation page. Included there are instructions to "
Collect the installer log file."
Collect SQL Server Information
To get a full picture of some problems, it's also useful to have information about running queries and locks. For this, you want both a
thread dump and information about the state of database connections. The latter needs to be obtained directly from SQL Server.
- Launch SQL Server Management Console or a similar tool
- Open a connection to the LabKey Server database, often named "labkey."
- Run and capture the output from the following queries/stored procedures:
- sp_lock
- sp_who2
- SELECT t1.resource_type, t1.resource_database_id, t1.resource_associated_entity_id, t1.request_mode, t1.request_session_id, t2.blocking_session_id FROM sys.dm_tran_locks as t1 INNER JOIN sys.dm_os_waiting_tasks as t2 ON t1.lock_owner_address = t2.resource_address;
Related Topics
