Dump Memory and Thread Usage Debugging Information

Documentation
To assist in debugging errors, an administrator can force the LabKey Server process to dump the list of threads and memory to disk. This information can then be sent to LabKey for review.

A memory/heap dump is useful for diagnosing issues where LabKey Server is running out of memory, or in some cases where the process is sluggish and consuming a lot of CPU.

A thread dump is useful for diagnosing issues where LabKey Server is hung or some requests spin forever in the web browser.

Definitions:

  • LABKEY_HOME: Installation location of the LabKey Server
    • If you performed a Manual Installation and followed these instructions, then this is most likely
      • /usr/local/labkey
    • If you used our Windows installer, then this is most likely one of the following:
      • C:\Program Files (x86)\LabKey Server\
      • C:\Program Files\LabKey Server\
  • CATALINA_HOME: Installation location of the Apache Tomcat Web Server
    • If you performed a Manual Installation and followed these instructions, then this is most likely
      • /usr/local/tomcat
    • If you used our Windows installer, then this is most likely one of the following:
      • C:\Program Files (x86)\LabKey Server\apache-tomcat-*.*.*
      • C:\Program Files\LabKey Server\apache-tomcat-*.*.*

Memory Dump

To request LabKey Server dump its memory, you can use either the web UI or the command line.

Using the web UI

  • Go to (Admin) > Site > Admin Console.
  • Select the Admin Console Links tab.
  • Under Diagnostics, click Dump Heap.
  • You will see a full path to where the heap was dumped.

Manually via the command line

Unix. On a Unix server (Linux, MacOSX, Solaris), execute the following commands on the command line:

  • Navigate to the LABKEY_HOME directory.
  • Force the server to dump its memory
touch heapDumpRequest

Windows. On a Windows Server, do the following:

  • Open a Command Prompt
  • Navigate to the LABKEY_HOME\bin directory.
  • Force the server to dump its memory. This command will open the heapDumpRequest file in the notepad program.
notepad heapDumpRequest
  • Place the cursor at the top of the file and hit the Enter key twice.
    • Save the file
    • Close notepad
Location of the file containing the memory dump

The file will be located in the LABKEY_HOME directory. The file will be named something like 'java_pidXXXX.hprof'.

Dump Running Threads

To request LabKey Server dump the list of running threads, use either the web UI or the command line.

Using the web UI

  • Go to (Admin) > Site > Admin Console.
  • Select the Admin Console Links tab.
  • Under Diagnostics, click Running Threads
  • This will show the state of all active threads in the browser, as well as writing the thread dump to the server log file

Manually via the command line

Unix. On a Unix server (Linux, MacOSX, Solaris), use the command line execute the following commands:

  • Change to the `LABKEY_HOME` directory. For example, if your 'LABKEY_HOME' directory is located at `/usr/local/labkey`, then the command will be
cd /usr/local/labkey
  • Force the server to dump its memory
touch threadDumpRequest

Windows. On a Windows Server, do the following:

  • Open a Command Prompt
  • Change to the `LABKEY_HOME\bin` directory. For example, if your 'LABKEY_HOME' directory is located at `C:\Program Files\LabKey Server\`, then the command will be
cd "C:\Program Files\LabKey Server"
  • Force the server to dump its memory. The command will open the heapDumpRequest file in the notepad program.
notepad threadDumpRequest
  • Place the cursor at the top of the file and hit the "Enter" key twice
    • Save the file
    • Close notepad
Location of the file containing the thread dump

The list of threads is dumped the `labkey.log` file, which is located in the `CATALINA_HOME\logs`

Debugging Postgres Errors

LabKey server is unable to log errors thrown by Postgres, so in the case of diagnosing some installation and startup errors, it may be helpful to to view the event log.

On Windows:
  • Launch eventvwr.msc
  • Navigate to Windows Logs > Application.
  • Search for errors there corresponding to the installation failures, which may assist LabKey support in diagnosing the problem.
  • If you can't find relevant messages, you may be able to trigger the error to occur again by running net start LabKey_pgsql-9.2 from the command line.

In addition, you may find assistance debugging PostgreSQL on their troubleshooting installation page. Included there are instructions to 'Collect the installer log file'.

Collect SQL Server Information

To get a full pictures of some problems, it's useful to have information about running queries and locks. For this, you want both a thread dump (described above) and information about the state of database connections. The latter needs to be obtained directly from SQL Server.

  • Launch SQL Server Management Console or a similar tool
  • Open a connection to the LabKey Server database, often named 'labkey'.
  • Run and capture the output from the following queries/stored procedures:
    • sp_lock
    • sp_who2
    • SELECT t1.resource_type, t1.resource_database_id, t1.resource_associated_entity_id, t1.request_mode, t1.request_session_id, t2.blocking_session_id FROM sys.dm_tran_locks as t1 INNER JOIN sys.dm_os_waiting_tasks as t2 ON t1.lock_owner_address = t2.resource_address;

Discussion

previousnext
 
expand all collapse all