Ben,

I am not of exactly how the code is written but I know that when the LabKey server or the LabKey remote pipeline server (ie what you have installed on gems) loses connectivity to the activeMQ server they do something like

1) Try to connect to the server for x number of times
2) wait 1 seconds try again
3) wait 5 seconds try again
4) wait 10 seconds try again
....

this will go on until eventually either the server stops attempting to reconnect or the time between reconnect attempts is large.

I believe frequency of the errors you are seeing from activeMQ will probably go down when you are able to clear up the perf problems on xnight. In the mean-time, I would recommend that if you have to restart the activeMQ process on xnight or reboot xnight, then restart the LabKey remote pipeline server on gems afterwards.

About your monit monitoring, please note that monit is simply checking if the activeMQ server is listening on the tcp port. This check is not sufficient to determine if activeMQ is fully operational (ie tasks will run on the pipeline). The current monitoring is designed to simply check if the activeMQ server has been started or not. To determine if the server is running and fully operational (ie you are able to put tasks on the queue and take them off, etc) then you will need add monitoring with tools like http://activemq.apache.org/how-can-i-monitor-activemq.html or http://it.toolbox.com/blogs/unix-sysadmin/monitoring-activemq-from-nagios-27743