Introduction
To assist in debugging the LabKey Server process, an administrator can force the LabKey Server process to the dump the list of threads and memory to disk. This information can then be sent to LabKey for review.
A memory/heap dump is useful for diagnosing issues where LabKey Server is running out of memory, or in some cases where the process is sluggish and consuming a lot of CPU.
A thread dump is useful for diagnosing issues where LabKey Server is hung or some requests spin forever in the web browser.
This can be done either through the LabKey Server web user interface or manually by touching a file on the web server.
Definitions:
- LABKEY_HOME: Installation location of the LabKey Server
- If you performed a Manual Installation and followed these instructions, then this is most likely
- If you used our Windows installer, then this is most likely one of the following:
- C:\Program Files (x86)\LabKey Server\
- C:\Program Files\LabKey Server\
- CATALINA_HOME: Installation location of the Apache Tomcat Web Server
- If you performed a Manual Installation and followed these instructions, then this is most likely
- If you used our Windows installer, then this is most likely one of the following:
- C:\Program Files (x86)\LabKey Server\apache-tomcat-*.*.*
- C:\Program Files\LabKey Server\apache-tomcat-*.*.*
Request LabKey Server dump its memory
This can be done using either the web UI or the command line.
Using the web UI
- Goto to the Admin Console for your server
- Click on the link dump heap
Manually via the command line
Unix. On a Unix server (Linux, MacOSX, Solaris), use the command line execute the following commands:
- Change to the `LABKEY_HOME` directory. For example, if your directory is located at /usr/local/labkey, then the command will be
- Force the server to dump its memory
Windows. On a Windows Server, do the following:
- Open a Command Prompt
- Start --> Programs --> Accessories --> Command Prompt
- Change to the `LABKEY_HOME\bin` directory. For example, if your 'LABKEY_HOME' directory is located at `C:\Program Files\LabKey Server\`, then the command will be
cd "C:\Program Files\LabKey Server"
- Force the server to dump its memory. The command will open the heapDumpRequest file in the notepad program.
- Place the cursor at the top of the file and hit the Enter key twice.
- Save the file
- Close notepad
Location of the file containing the memory dumpThe file will be located in the LABKEY_HOME directory. The file will be named something like 'java_pidXXXX.hprof'.
Request LabKey Server dump the list of running threads
This can be done using either the web UI or the command line.
Using the web UI
- Goto to the Admin Console for your server
- Click on the link running threads
- This will show the state of all active threads in the browser, as well as writing the thread dump to the server log file
Manually via the command line
Unix. On a Unix server (Linux, MacOSX, Solaris), use the command line execute the following commands:
- Change to the `LABKEY_HOME` directory. For example, if your 'LABKEY_HOME' directory is located at `/usr/local/labkey`, then the command will be
- Force the server to dump its memory
Windows. On a Windows Server, do the following:
- Open a Command Prompt
- Start --> Programs --> Accessories --> Command Prompt
- Change to the `LABKEY_HOME\bin` directory. For example, if your 'LABKEY_HOME' directory is located at `C:\Program Files\LabKey Server\`, then the command will be
cd "C:\Program Files\LabKey Server"
- Force the server to dump its memory. The command will open the heapDumpRequest file in the notepad program.
notepad threadDumpRequest
- Place the cursor at the top of the file and hit the "Enter" key twice
- Save the file
- Close notepad
Location of the file containing the thread dumpThe list of threads is dumped the `labkey.log` file, which is located in the `CATALINA_HOME\logs`
Gather debugging information about Postgres errors
LabKey server is unable to log errors thrown by Postgres, so in the case of diagnosing some installation and startup errors, it may be helpful to to view the event log.
On Windows:
- Launch eventvwr.msc
- Navigate to Windows Logs > Application.
- Search for errors there corresponding to the installation failures, which may assist LabKey support in diagnosing the problem.
- If you can't find relevant messages, you may be able to trigger the error to occur again by running net start LabKey_pgsql-9.2 from the command line.
In addition, you may find assistance debugging PostgreSQL on their
troubleshooting installation page. Included there are instructions to 'Collect the installer log file'.
Collect SQL Server information about running queries and locks
To get a full pictures of some problems, it's useful to have both a thread dump (described above) and information about the state of database connections. The latter needs to be obtained directly from SQL Server.
- Launch SQL Server Management Console or a similar tool
- Open a connection to the LabKey Server database, often named 'labkey'.
- Run and capture the output from the following queries/stored procedures:
- sp_lock
- sp_who2
- SELECT t1.resource_type, t1.resource_database_id, t1.resource_associated_entity_id, t1.request_mode, t1.request_session_id, t2.blocking_session_id FROM sys.dm_tran_locks as t1 INNER JOIN sys.dm_os_waiting_tasks as t2 ON t1.lock_owner_address = t2.resource_address;
