One of our developers encountered the following very disturbing behavior on our production system:

"I was making some changes to a parameterized query that is used in a javascript wiki page “report”. I had a few labkey tabs open, and was going back and forth between the query, the wiki page, and the main dataset being queried (Adverse Events). I returned to the query to make an additional change and I got a message that there were errors in the query, because the Adverse Events dataset could not be found. I figured this was just some sort of glitch since I knew the dataset was there, and so I went ahead and made a change to the query and saved it. The query wouldn’t run due to the same error about not finding the dataset, so I went to the main project to take a look at the datasets. There should be 56. The Adverse Events dataset was not in the list, and I noticed that some other datasets were missing as well, and many of the datasets that were there were listed twice. I counted a total of 41 unique datasets (I didn’t count how many doubles there were). I believe that most of the datasets that disappeared were in the bottom half of my dataset list, but it wasn’t the last 15 or anything that straightforward."

A couple others of us visited the same Study main page from our own machines/browsers, and saw the same wrong list of datasets.

Re-starting Tomcat cleared up the problem, but obviously this behavior is a serious concern to us.

I have attached a portion of our labkey.log file from near the time this occurred. Note the exceptions that are reported starting at 2013-06-19 11:09:56,412. Re-start begins around 2013-06-19 11:48:49,155.

We are running LabKey 13.1 revision 25846 on RHEL 5.7, Tomcat 6.0.36, PostgreSQL 8.4.9, and Java 1.7.0_15.

Can anyone make any sense of this, or have any idea what the cause could be, or what we can do to prevent this happening?

Thanks.

Andy Straw
University of Rochester