CPAS / Mascot Timeout

CPAS Forum (Inactive)
CPAS / Mascot Timeout gregb  2007-01-23 12:47
Status: Closed
Assigned To: adam
 
I performed a manual install of CPAS on CentOS 4.4 and I am able to use the local X!Tandem server (on the same machine) with no problems. However, when I am trying to connect to my Windows 2003 server based Mascot server, every search times out after roughly 1 minute. Nothing is showing up in the cpas-error log, and on the Mascot server side, I can watch the nph-mascot.exe process run until it is killed off. The only log on the windows side that shows any information is C:\WINDOWS\system32\LogFiles\HTTPERR\httperr1.log which only shows the following:
2007-01-23 20:39:23 138.26.53.69 11132 138.26.53.67 80 HTTP/1.1 POST /mascot/cgi/nph-mascot.exe?1+--taskID+116958469701+--sessionID+all_secdisabledsession - 1 Connection_Dropped DefaultAppPool

I have all of my timeouts on the IIS side set to a minimum of atleast 20 minutes, including the cgi timeout, and I don't notice any timeouts on the Tomcat side that are anywhere near that low either. So any ideas on what timeout I am missing? The error log for the job inside CPAS is showing a mascot error of -1.

Thanks,
Greg
 
 
adam responded:  2007-01-23 18:01
Assigned To: wongch
 
gregb responded:  2007-01-24 12:43
After trying to grep for timeout in the source files, I came upon this:
caBIG/axis-1_4/changelog.html: <li> The default socket timeout for clients is now 60 seconds, not infinity.

I'm not sure if axis is being used during the wait for mascot to return a value or if it is using the default. Without being familiar with the source code, or java, that might be a place to start looking for the issue.
 
wongch responded:  2007-01-24 16:59
Hi Greg,

Thanks for the pointer. This part of the code uses HTTPClient. Thanks for checking out the similar symptom case. (See https://cpas.fhcrc.org/announcements/home/support/thread.view?rowId=963)

If you are using v1.7, do you mind to pick up "api.jar" from my posting dated "Wed Jan 10 18:33:47 PST 2007" at the thread mentioned above and try out with a small file "CAexample_mini.mzXML". (Also found in the same thread.) Please try out and provide your search log.

If you do not mind, please help with the followings:

1) Send the followings 3 mascot logs: /mascot/logs/errorlog.txt, /mascot/logs/security.log, /mascot/logs/searches.log ?

2) Do you mind to point your browser on the CPAS server to your Mascot server? Navigate to the MS/MS search page and submit the CAexample_mini.mgf generated in the by CPAS for the search? (The file will be in the subfolder of the pipeline root.) Let us know how this search goes.

Cheers.
 
gregb responded:  2007-01-24 18:16
I'll try that in the morning. I don't have time to try it from home tonight.

As for the .mzxml file, I was actually just using the Example data set and was trying to run it against either Swissprot or Uniref100. With Swissprot I was getting 5% processed before it was killed. I'll do your tests tomorrow morning and upload the results.

Greg
 
gregb responded:  2007-01-25 07:05
Ok, I ran all of the tests that you wanted, and in addition to the logs you requested, I added the cpas logs for extra information. I reset all of the logfiles before starting the session, so you have the logs only for tests. The mascot logs (which are basically empty) also include the direct run of the .mgf file, which ran to completion. The CAexample_mini search ran to completion from CPAS, but crashed right afterward, which is evident in the log files. I reran the MiniICAT run so the error would show in the logs, and that was the same file I ran via the .mgf file.

Hopefully this extra data will give you everything that you need to pinpoint the problem.

Greg
 
wongch responded:  2007-01-25 07:50
Thanks, Greg. Can I trouble you for
1) /home/cpas/CAexample/CPAS/mascot/Mascot Test/CAexample_mini.log and
2) /home/cpas/ExampleFiles/CPAS/mascot/Test Mascot2/SmallICAT.log ?
*Chee-Hong
 
gregb responded:  2007-01-25 08:21
No problem at all. Here are the files.
 
wongch responded:  2007-01-26 09:32
Greg, to move thing forward, I have added re-try for the status retrieval and a few more logging. Please replace api.jar and try both your small searches. Please post the CPAS search log (CAexample_mini.log and SmallICAT.log), cpas.log and cpas-error.log. Thanks.
 
gregb responded:  2007-01-26 09:53
Done, here are the logs.
 
wongch responded:  2007-02-01 01:30
Hi Greg and John,

Sorry for the long delay. I have compiled a new api.jar. Please drop it in <cpaswebapp dir>\WEB-INF\lib\ after backup the original copy there. Restart your tomcat and try out CAexample_mini.mzXML and then SmallICAT.mzXML.

If it still does not work out, please post your search log file, cpas.log and cpas-error.log here. Thanks.

*Chee-Hong.
 
gregb responded:  2007-02-01 07:31
The search completed successfully, so it looks like the new change fixed that problem. Once the job was done, I did get a java error though. You probably saw it in the logs I sent previously for the CAexample run, but here is the error again:
01 Feb 2007 09:28:34,086 ERROR: Uncaught exception in PiplineJob: 1(DONE) CPAS/SmallICAT (Test_ICAT3)
java.lang.NullPointerException
    at org.fhcrc.cpas.ms2.pipeline.MascotPipelineJob.search(MascotPipelineJob.java:332)
    at org.fhcrc.cpas.ms2.pipeline.AbstractMS2SearchPipelineJob.run(AbstractMS2SearchPipelineJob.java:180)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
    at java.util.concurrent.FutureTask.run(FutureTask.java:123)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:65)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:168)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
    at java.lang.Thread.run(Thread.java:595)

Let me know if you want the logs again, since you should have some traces in the previous logs.

Thanks for the help,
Greg
 
wongch responded:  2007-02-01 08:05
Hi Greg,

I will like to have your search log file, cpas.log and cpas-error.log for both mzXML files search. Please also help me by providing the zipped up of subfolder "CAexample_mini.work" and "SmallICAT.work" under the pipeline root directory. Finally, please provide a directory listing of the fasta root that you have configured.

Thanks.
*Chee-Hong
 
gregb responded:  2007-02-01 08:26
The searches were done on the Swissprot database (Sprot)

ls -al /home/cpas/databases/
total 2201220
drwxrwxrwx 2 root root 4096 Jan 23 14:34 .
drwxrwxrwx 6 root root 4096 Jan 25 08:37 ..
-rwxrwxrwx 1 root root 110529 May 24 2006 150Proteins.fsa
-rwxrwxrwx 1 root root 3958 May 24 2006 7Proteins.fsa
-rwxr-xr-x 1 root root 123471014 Jan 23 14:37 Sprot
-rwxrwxrwx 1 root root 2128234565 Jan 23 14:20 UniRef100

I've attached the rest of the files you requested.
 
wongch responded:  2007-02-01 09:26
Thanks. Please drop this new api.jar in-place and just try out "CAexample_mini.mzXML" alone. Please post the search log file "CAexample_mini.log". This new jar just dumps out the various variables in the composite statement that has the null pointer exception. Hope this shed light on the cause.
 
gregb responded:  2007-02-01 09:32
Here you go.

Greg
 
wongch responded:  2007-02-01 09:46
Have you set your "FASTA root" for the pipeline? Select your project on the left panel menu. Click on the "Pipeline tab", then "Setup" button. Click on link "Set FASTA root" under the section "MS2 specific settings:". Is the entry value "/home/cpas/databases/" ?
 
gregb responded:  2007-02-01 09:54
There isn't a Setup Button once the project is selected, but from the main page, it is defined as /home/cpas/databases (the trailing / is removed when it is set). My cpas home is also set to /home/cpas, just in case that helps. Also, as I said before, using X!Tandem works fine, so it is something specific to Mascot.

Greg
 
wongch responded:  2007-02-01 10:30
Do you mind to give this new api.jar a shot? Please post the search log. Your point on X!Tandem is logical. Thanks!
 
gregb responded:  2007-02-01 10:42
That worked, but now there is a problem during the database loading. I've attached the search log.

Greg
 
wongch responded:  2007-02-01 10:53
Please post the zipped content of the subfolder "CAexample_mini.work" for this latest search. I will look thru' them later this morning.

By the way, do you have administrator right on this project folder? You should see a bar of buttons "[Process and Upload Data] [Setup] [Select All] [Clear All] [Delete] [Complete]" at the end of page when you click on the "Pipeline" tab. This [Setup] button will bring you to the page for setting up FASTA root.
 
gregb responded:  2007-02-01 11:02
The CAexample_mini.work folder is not in the directory any more. I am assuming that it was deleted when the database upload was attempted. I am including a zip of /home/cpas/CAexample/CPAS/mascot/Mascot_Test5, since that should have everything that you need.

My account has full admin rights, as I am setup as the site admin, but in the project folder the setup tab isn't there. I included a jpeg screen shot to show you what I am looking at.

Thanks again for the help,
Greg
 
brendanx responded:  2007-02-01 15:03
Looks like a prolem in Mascot2XML encoding the ampersand character. The problem line is:

<parameter name="LICENSE" value="Licensed to: Univ. of Alabama at Birmingham, Pharmacology & Toxicology (106936), (1 processor)."/>

That ampersand needs to be &amp; to work in XML.

Chee Hong can probably fix this in the TPP.
 
wongch responded:  2007-02-02 07:32
Hi Greg,

The fix has been committed to Sashimi CVS. Please pick up the new content in trans_proteomic_pipeline/src/Parsers/Algorithm2XML/Mascot2XML and build Mascot2XML.

*Chee-Hong.
 
gregb responded:  2007-02-02 14:56
Sorry I delayed the testing due to a mistake on my part earlier, but the data now loads into the database successfully. It looks like everything works fine now.

Thanks for all of the help.
Greg
 
wongch responded:  2007-02-04 17:15
Thanks, Greg for working together to nail down and resolve this issue.

To summarize:

1. If your Mascot search log indicate a "error=-1" message after submitting the search, please try out api.jar in message dated Thu Feb 01 10:30:55 PST 2007. (Drop this jar file into <cpaswebapp dir>\WEB-INF\lib\)

2. If your Mascot licensee contain any of these special characters & " ' < >
   you need to build a new copy of Mascot2XML. Download the code from Sashimi website. Ensure that /trans_proteomic_pipeline/src/Parsers/Algorithm2XML/Mascot2XML/MascotConverter.cxx is v1.51 or higher.

Note that these 2 issues are independent.

*Chee-Hong.