• Sign In
  1. Messages

Postgres I/O problems after upgrade to 10.3

Installation Forum (Inactive)

View Message

view list print
Postgres I/O problems after upgrade to 10.3 mboehmer  2010-12-06 15:03
Status: Closed
 
Hi there,

I experienced a problem doing X!tandem searches after upgrade to 10.3. The log file of the corresponding search indicates a Postgresql I/O error. I am running Postgresql 8.3. This has happened a few times now, the server stops corresponding and only restart of the computer helps. Any ideas how to solve the problem: install newer postgresql, roll back the Labkey upgrade, ...?

Thanks for advice,

Maik


03 Dec 2010 09:19:43,390 INFO : Starting to run task 'org.labkey.ms2.pipeline.tandem.XTandemSearchTask' at location 'webserver'
03 Dec 2010 09:19:44,562 INFO : tandem.exe output
03 Dec 2010 09:19:44,562 INFO : =======================================
03 Dec 2010 09:19:44,562 INFO : Working directory is E:\MB20101119\xtandem\Tair9_14N_IAA_quant_v104\MB20101116_A12-s1.work
03 Dec 2010 09:19:44,562 INFO : running: C:\Program Files\LabKey Server\bin\tandem.exe input.xml
03 Dec 2010 09:19:46,171 INFO :
03 Dec 2010 09:19:46,171 INFO : X! TANDEM 2009.10.01.1 (LabKey and ISB)
03 Dec 2010 09:19:46,171 INFO :
03 Dec 2010 09:20:08,812 INFO : Loading spectra .... loaded.
03 Dec 2010 09:20:08,812 INFO : Spectra matching criteria = 6904
03 Dec 2010 09:20:08,812 INFO : Pluggable scoring enabled.
03 Dec 2010 09:20:08,859 INFO : Starting threads .. started.
03 Dec 2010 09:20:08,859 INFO : Computing models:
03 Dec 2010 10:54:13,421 INFO :     Spectrum-to-sequence matching proc
03 Dec 2010 10:54:13,421 INFO :     waiting for 2 done.
03 Dec 2010 10:54:13,421 INFO :
03 Dec 2010 10:54:13,421 INFO :     sequences modelled = 33 ks
03 Dec 2010 10:54:13,484 INFO : Model refinement:
03 Dec 2010 10:54:32,656 INFO :     partial cleavage ...... done.
03 Dec 2010 14:04:20,953 INFO : Failed to complete task 'org.labkey.ms2.pipeline.tandem.XTandemSearchTask'
03 Dec 2010 14:04:22,734 ERROR: Failed to remove work directory E:\MB20101119\xtandem\Tair9_14N_IAA_quant_v104\MB20101116_A12-s1.work unexpected files found:
input.xml
taxonomy.xml
03 Dec 2010 14:04:31,328 ERROR: Failed to set status to 'ERROR' for 'E:\MB20101119\xtandem\Tair9_14N_IAA_quant_v104\MB20101116_A12-s1.log'.
org.postgresql.util.PSQLException: An I/O error occured while sending to the backend.
    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:283)
    at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
    at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
    at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273)
    at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:93)
    at org.labkey.api.data.StatementWrapper.executeQuery(StatementWrapper.java:543)
    at org.labkey.api.data.Table._executeQuery(Table.java:146)
    at org.labkey.api.data.Table._executeQuery(Table.java:111)
    at org.labkey.api.data.Table.internalExecuteQueryArray(Table.java:356)
    at org.labkey.api.data.Table.select(Table.java:1324)
    at org.labkey.api.data.Table.select(Table.java:1299)
    at org.labkey.api.data.Table.select(Table.java:1291)
    at org.labkey.pipeline.api.PipelineStatusManager.getStatusFile(PipelineStatusManager.java:115)
    at org.labkey.pipeline.api.PipelineStatusManager.getJobStatusFile(PipelineStatusManager.java:90)
    at org.labkey.pipeline.api.PipelineStatusManager.setStatusFile(PipelineStatusManager.java:128)
    at org.labkey.pipeline.api.PipelineServiceImpl.setStatusFile(PipelineServiceImpl.java:435)
    at org.labkey.api.pipeline.PipelineJob.setStatus(PipelineJob.java:425)
    at org.labkey.api.pipeline.PipelineJob$OutputLogger.setErrorStatus(PipelineJob.java:1183)
    at org.labkey.api.pipeline.PipelineJob$OutputLogger.error(PipelineJob.java:1146)
    at org.labkey.api.pipeline.PipelineJob.error(PipelineJob.java:1322)
    at org.labkey.api.pipeline.PipelineJob.error(PipelineJob.java:1315)
    at org.labkey.api.pipeline.PipelineJob.runActiveTask(PipelineJob.java:612)
    at org.labkey.api.pipeline.PipelineJob.run(PipelineJob.java:813)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Unexpected packet type: 0
    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1981)
    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
    ... 30 more
03 Dec 2010 14:04:35,656 ERROR: Uncaught exception in PiplineJob: (DONE) MB20101119/MB20101116_A12-s1 (Tair9_14N_IAA_quant_v104)
org.postgresql.util.PSQLException: An I/O error occured while sending to the backend.
    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:283)
    at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
    at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
    at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273)
    at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:93)
    at org.labkey.api.data.StatementWrapper.executeQuery(StatementWrapper.java:543)
    at org.labkey.api.data.Table._executeQuery(Table.java:146)
    at org.labkey.api.data.Table._executeQuery(Table.java:111)
    at org.labkey.api.data.Table.internalExecuteQueryArray(Table.java:356)
    at org.labkey.api.data.Table.select(Table.java:1324)
    at org.labkey.api.data.Table.select(Table.java:1299)
    at org.labkey.api.data.Table.select(Table.java:1291)
    at org.labkey.pipeline.api.PipelineStatusManager.getStatusFile(PipelineStatusManager.java:115)
    at org.labkey.pipeline.api.PipelineStatusManager.getJobStatusFile(PipelineStatusManager.java:90)
    at org.labkey.pipeline.api.PipelineStatusManager.setStatusFile(PipelineStatusManager.java:128)
    at org.labkey.pipeline.api.PipelineServiceImpl.setStatusFile(PipelineServiceImpl.java:435)
    at org.labkey.api.pipeline.PipelineJob.setStatus(PipelineJob.java:425)
    at org.labkey.api.pipeline.PipelineJob$OutputLogger.setErrorStatus(PipelineJob.java:1183)
    at org.labkey.api.pipeline.PipelineJob$OutputLogger.error(PipelineJob.java:1146)
    at org.labkey.api.pipeline.PipelineJob.error(PipelineJob.java:1322)
    at org.labkey.api.pipeline.PipelineJob.error(PipelineJob.java:1315)
    at org.labkey.api.pipeline.PipelineJob.runActiveTask(PipelineJob.java:612)
    at org.labkey.api.pipeline.PipelineJob.run(PipelineJob.java:813)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Unexpected packet type: 0
    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1981)
    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
    ... 30 more
 
 
jeckels responded:  2010-12-07 17:39
Hi Maik,

A few questions about your setup:

Is your database running on the same machine as the web server? If not, is it also running on Windows?

Have you changed Postgres versions as part of the upgrade, or any time recently?

Does it always seem to be related to hitting an error when doing an X!Tandem search, or have you seen it at times when no searches were running?

Thanks,
Josh
 
mboehmer responded:  2010-12-08 13:03
Hi,

The database is indeed running on the same machine as the web server. Postgres was not upgraded, we have used the 8.3 version already under Labkey 10.2. I have also seen the web server crash when running the XpressPeptideParser. Then it shows an unexpected error related to the JAVA heap dump or some memory error. I attach the Apache error output I have seen in these cases:

Thanks for your help

message

description The server encountered an internal error () that prevented it from fulfilling this request.

exception

javax.servlet.ServletException: Servlet execution threw an exception
    org.labkey.api.data.TransactionFilter.doFilter(TransactionFilter.java:36)
    org.labkey.core.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:118)
    org.labkey.api.module.ModuleLoader.doFilter(ModuleLoader.java:603)
    org.labkey.api.security.AuthFilter.doFilter(AuthFilter.java:138)

root cause

java.lang.OutOfMemoryError: Java heap space

note The full stack trace of the root cause is available in the Apache Tomcat/5.5.27 logs.
 
Brian Connolly responded:  2010-12-08 15:30
Maik,

For some reason, it looks like apache log file (labkey.log) did not get attached with your support board posting. Can you send it again.

About the Java Heap space message. I recommend increasing the tomcat configuration to use more heap space. You can follow these instructions https://www.labkey.org/wiki/home/Documentation/page.view?name=configWebappMemory

If you have ~4GB of memory on the server, I would recommend setting the Maximum memory pool to be 4GB. If you have 2GB of memory on the server, I recommend setting the Maximum memory pool to 1GB.


Thank you,

Brian
 
mboehmer responded:  2010-12-09 07:49
Hi Brian,

thanks for the link to change the JAVA heap space. Method 2 worked for me and the server seems to be running stable now. The other thing I did was to move my mzXML files for the tandem searches from an external hard drive back to the C drive. Not sure, this made any difference or whether it was only the heap space, but searches are running again without error messages.

Thank you,

Maik