Dump Memory and Thread Usage Debugging Information: /Documentation/Archive/19.3

Dump Memory and Thread Usage Debugging Information

To assist in debugging errors, an administrator can force the LabKey Server process to dump the list of threads and memory to disk. This information can then be sent to LabKey for review.

A thread dump is useful for diagnosing issues where LabKey Server is hung or some requests spin forever in the web browser.

A memory/heap dump is useful for diagnosing issues where LabKey Server is running out of memory, or in some cases where the process is sluggish and consuming a lot of CPU performing garbage collection.

Thread Dump
Memory Dump
Debugging PostgreSQL Errors
Collect SQL Server Information

Definitions:

<LABKEY_HOME>: Installation location of the LabKey Server, for example:

Linux: /usr/local/labkey/labkey
Windows: C:\labkey\labkey\

<CATALINA_HOME>: Installation location of the Apache Tomcat Web Server (where x.x.xx is the specific version installed, for example:

Linux: /usr/local/labkey/apps/apache/apache-tomcat-x.x.xx
Windows: C:\labkey\apps\apache\apache-tomcat-x.x.xx

Thread Dump

To dump the list of running threads, you can use either the UI or the command line.

Using the UI

Go to (Admin) > Site > Admin Console.
Click the Settings tab.
Under Diagnostics, click Running Threads
This will show the state of all active threads in the browser, as well as writing the thread dump to the server log file.

Manually via the Command Line

The server monitors the timestamp of a threadDumpRequest file and when it notices a change, it initiates the thread dump process. The content of the file does not matter.

Unix. On Linux and OSX, use the command line to execute the following commands:

Change to the <LABKEY_HOME> directory. For example, if your <LABKEY_HOME> directory is located at /usr/local/labkey/labkey, then the command will be:
```
cd /usr/local/labkey/labkey
```
Force the server to dump its memory:
```
touch threadDumpRequest
```

Windows. On a Windows Server, do the following:

Open a Command Prompt.
Change to the <LABKEY_HOME> directory. For example, if your <LABKEY_HOME> directory is located at C:\labkey\labkey\, then the command will be:

cd "C:\labkey\labkey"

Force the server to dump its memory. The command will open the heapDumpRequest file in the notepad program.

notepad threadDumpRequest

Place the cursor at the top of the file and hit the "Enter" key twice.

Save the file.
Close notepad.

Location of the file containing the thread dump

The list of threads is dumped the "labkey.log" file, which is located in the <CATALINA_HOME>/logs directory.

Memory Dump

To request LabKey Server dump its memory, you can use either the UI or the command line. By default, it will write the file to the LABKEY_HOME directory. Java can also be configured to dump the heap if the virtual machine runs out of memory.

Heap dumps can be large: several gigabytes in size is typical. You can tell the server to write to a different location by using a JVM startup argument,

-XX:HeapDumpPath=/path/to/desired/target

Using the UI

Go to (Admin) > Site > Admin Console.
Click the Settings tab.
Under Diagnostics, click Dump Heap.
You will see a full path to where the heap was dumped.

Manually via the Command Line

The server monitors the timestamp of a heapDumpRequest file and when it notices a change, it initiates the heap dump process. The content of the file does not matter.

Unix. On Linux and OSX, execute the following commands on the command line:

Navigate to the <LABKEY_HOME> directory.
Force the server to dump its memory

touch heapDumpRequest

Windows. On a Windows Server, do the following:

Open a Command Prompt
Navigate to the <LABKEY_HOME> directory.
Force the server to dump its memory. This command will open the heapDumpRequest file in the notepad program.

notepad heapDumpRequest

Place the cursor at the top of the file and hit the Enter key twice.

Save the file
Close notepad

Location of the file containing the memory dump

The file will be located in the <LABKEY_HOME> directory. The file will have the ".hprof" extension.

Automatically Capturing Heap Dump when Server Runs Out of Memory

The Java virtual machine supports an option that will automatically trigger a heap dump when the server runs out of memory. This is accomplished via this JVM startup parameter:

-XX:+HeapDumpOnOutOfMemoryError

For more information, see Oracle's documentation.

Debugging PostgreSQL Errors

LabKey server is unable to log errors thrown by PostgreSQL, so in the case of diagnosing some installation and startup errors, it may be helpful to to view the event log.

On Windows:

Launch eventvwr.msc
Navigate to Windows Logs > Application.
Search for errors there corresponding to the installation failures, which may assist LabKey support in diagnosing the problem.
If you can't find relevant messages, you may be able to trigger the error to occur again by running net start LabKey_pgsql-9.2 from the command line.

In addition, you may find assistance debugging PostgreSQL on their troubleshooting installation page. Included there are instructions to "Collect the installer log file."

Collect SQL Server Information

To get a full picture of some problems, it's also useful to have information about running queries and locks. For this, you want both a thread dump (described above) and information about the state of database connections. The latter needs to be obtained directly from SQL Server.

Launch SQL Server Management Console or a similar tool
Open a connection to the LabKey Server database, often named "labkey."
Run and capture the output from the following queries/stored procedures:

sp_lock
sp_who2
SELECT t1.resource_type, t1.resource_database_id, t1.resource_associated_entity_id, t1.request_mode, t1.request_session_id, t2.blocking_session_id FROM sys.dm_tran_locks as t1 INNER JOIN sys.dm_os_waiting_tasks as t2 ON t1.lock_owner_address = t2.resource_address;

LabKey Support

LabKey Support