error after restart labkey/tomcat

General Server Forum (Inactive)
error after restart labkey/tomcat toan nguyen  2017-11-29 12:34
Status: Closed
 
Hi,

    Initially I have "transport is not running" which seems to point to activemq
    I found out that activemq is not running so I started it. However; it still does not work.
    So I restart labkey/tomcat however; tomcat seem taking forever to load and labkey seem to report the error below.
 
   Would you please point me to the right direction to fix this ?
   I am running labkey 15 on ubuntu 12.10.

Thanks
Toan.

INFO MuleManager 2017-11-29 12:05:57,495 Module Starter : Connectors have been started successfully
INFO MuleManager 2017-11-29 12:05:57,495 Module Starter : Starting agents ...
INFO MuleManager 2017-11-29 12:05:57,495 Module Starter : Agents Successf ully Started
ERROR eRetryConnectionStrategy 2017-11-29 12:06:12,307 UMOManager.4 : Failed to connect/reconnect: ActiveMqJmsConnector{this=3c46d6e5, started=false, initialised=true, name='jmsConnectorCloud', disposed=false, numberOfConcurrentTransactedReceivers=4, createMultipleTransactedReceivers=true, connected=true, supportedProtocols=[jms], serviceOverrides=null}. Root Exception was: Failed to start Jms Connection. Type: class org.mule.umo.lifecycle.LifecycleException
org.mule.umo.lifecycle.LifecycleException: Failed to start Jms Connection
        at org.mule.providers.jms.JmsConnector.doStart(JmsConnector.java:504)
        at org.mule.providers.AbstractConnector.startConnector(AbstractConnector.java:367)
        at org.mule.providers.AbstractConnector.connect(AbstractConnector.java:1186)
        at org.mule.providers.SimpleRetryConnectionStrategy.doConnect(SimpleRetryConnectionStrategy.java:76)
        at org.mule.providers.AbstractConnectionStrategy$1.run(AbstractConnectionStrategy.java:57)
        at org.mule.impl.work.WorkerContext.run(WorkerContext.java:310)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1061)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)
        at java.lang.Thread.run(Thread.java:745)
 
 
toan nguyen responded:  2017-11-29 12:39
It is taking a toll on the server , making the server non-responsive.
Will the process time out eventually ?

ERROR eRetryConnectionStrategy 2017-11-29 12:06:12,307 UMOManager.4 : Failed to connect/reconnect: ActiveMqJmsConnector{this=3c46d6e5, started=false, initialised=true, name='jmsConnectorCloud', disposed=false, numberOfConcurrentTransactedReceivers=4, createMultipleTransactedReceivers=true, connected=true, supportedProtocols=[jms], serviceOverrides=null}. Root Exception was: Failed to start Jms Connection. Type: class org.mule.umo.lifecycle.LifecycleException
org.mule.umo.lifecycle.LifecycleException: Failed to start Jms Connection
        at org.mule.providers.jms.JmsConnector.doStart(JmsConnector.java:504)
        at org.mule.providers.AbstractConnector.startConnector(AbstractConnector.java:367)
        at org.mule.providers.AbstractConnector.connect(AbstractConnector.java:1186)
        at org.mule.providers.SimpleRetryConnectionStrategy.doConnect(SimpleRetryConnectionStrategy.java:76)
        at org.mule.providers.AbstractConnectionStrategy$1.run(AbstractConnectionStrategy.java:57)
        at org.mule.impl.work.WorkerContext.run(WorkerContext.java:310)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1061)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)
        at java.lang.Thread.run(Thread.java:745)
 
toan nguyen responded:  2017-11-29 15:16
okay, stop all processes and restart
- activemq
- tomcat/labkey

labkey log show

 INFO ModuleLoader 2017-11-29 14:03:51,193 catalina-exec-11 : Starting up modules
INFO ModuleLoader 2017-11-29 14:03:51,196 Module Starter : Starting module 'Core'
INFO PipelineService 2017-11-29 14:03:51,421 UpgradeCheck : Found JMS queue; running Enterprise Pipeline.

Does that means labkey is up and running the pipeline job which was queued previously ?
That job is still running and taking all the server resources. The labkey webapp is not able to response . Is that normal behavior ?
After 1 hour, tomcat crash and restart, the log is attached.

Please help

Thanks
Toan
 
toan nguyen responded:  2017-11-29 15:46
I'm confused.
After awhile, labkey startup, but the webapp show site in maintenance mode and have the site admin login button. When I click on it, it just in spining around.
can someone help ?

Thanks
Toan



INFO ModuleLoader 2017-11-29 15:28:43,868 localhost-startStop-1 :
   __
   || | _ |_ |/ _ (¯ _ _ _ _
  (__) |_(_||_)|\(/_\/ _)(/_| \/(/_|
                    /
WARN PostgreSqlDialectFactory 2017-11-29 15:28:46,793 localhost-startStop-1 : LabKey Server no longer supports PostgreSQL version 9.1.14. PostgreSQL 9.4 is the recommended version.
INFO DbScope 2017-11-29 15:28:46,794 localhost-startStop-1 : Initializing DbScope with the following configuration:
    DataSource Name: labkeyDataSource
    Server URL: jdbc:postgresql://localhost/labkey
......
INFO MuleManager 2017-11-29 15:39:54,548 Module Starter : Connectors have been initialised successfully
INFO MuleManager 2017-11-29 15:39:54,550 Module Starter : Initialising agents...
INFO MuleManager 2017-11-29 15:39:54,555 Module Starter : Agents Successfully Initialised
INFO lineJobExceptionStrategy 2017-11-29 15:39:54,557 Module Starter : Initialising exception listener: org.labkey.pipeline.mule.PipelineJobExceptionStrategy@49b3f6c1
INFO lineJobExceptionStrategy 2017-11-29 15:39:54,560 Module Starter : Initialising exception listener: org.labkey.pipeline.mule.PipelineJobExceptionStrategy@2bd7044d
INFO MuleManager 2017-11-29 15:39:54,561 Module Starter : Server endpointUri is null, not registering Mule Admin agent
 
toan nguyen responded:  2017-11-29 16:21
so it run for awhile and activemq process die (see attachment)
Please help.
 
toan nguyen responded:  2017-11-29 16:28
I am not sure what is hanging.
When login the webapp, it shows.


Startup in progress    
Startup in progress: only site admins may login at this time. Your browser will be redirected when startup is complete
 
toan nguyen responded:  2017-11-29 16:40
this is going to a loop.
start tomcat/labkey,

>> INFO PipelineService 2017-11-29 16:35:35,408 UpgradeCheck : Found JMS queue; running Enterprise Pipeline.

webapp show maintenance mode.
Wait for ~hours, tomcat/java die/restart a few times, but the webapp still show maintenance mode.
restart tomcat/labkey again

>> INFO PipelineService 2017-11-29 16:35:35,408 UpgradeCheck : Found JMS queue; running Enterprise Pipeline.

what to do ?

Thanks
Toan
 
Jon (LabKey DevOps) responded:  2017-12-04 16:58
Hi Toan,

Please utilize the directions that Josh provided here:

https://www.labkey.org/announcements/home/Support/Support%20Forum/thread.view?rowId=16735

=======

Shut down Tomcat, and then shut down ActiveMQ. Ensure that both have fully exited (via a "ps" command on Linux or Task Manager on Windows, making sure there are no "java" processes running. Also shut down any LabKey remote pipeline servers that might be connected to ActiveMQ on other servers.

Restart ActiveMQ.

Restart Tomcat.

Restart the LabKey remote pipeline servers (if applicable).

The server should automatically requeue jobs that were in flight prior to the problems. If they don't move, try submitting a new job and see if it makes progress.

If things still aren't working correctly, please post the labkey.log file from the web server's TOMCAT/logs directory.
=======

Regards,

Jon