I have a cronjob that makes queries and then performs row deletions on lists using the LabKey API (via the python package). The queries are simple but can return a large number of rows. The deletions are batched to 900 rows per API call. This workflow has been running fine for a while.

Last week I upgraded from LabKey v23.3.2 to v23.7.0 and immediately started getting out of memory errors when performing the row deletions.

ERROR ExceptionUtil            2023-08-27T01:52:59,521     http-nio-8080-exec-2 : Unhandled exception: Java heap space
java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3537) ~[?:?]
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:228) ~[?:?]
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:582) ~[?:?]
	at java.lang.StringBuilder.append(StringBuilder.java:179) ~[?:?]
	at org.labkey.list.model.ListManager$2.exec(ListManager.java:844) ~[list-23.7.0.jar:?]
	at org.labkey.list.model.ListManager$2.exec(ListManager.java:840) ~[list-23.7.0.jar:?]
	at org.labkey.api.data.TableSelector.lambda$forEachResults$1(TableSelector.java:278) ~[api-23.7.0.jar:?]
	at org.labkey.api.data.TableSelector$$Lambda$1265/0x00007fc3d0e3c130.handle(Unknown Source) ~[?:?]
	at org.labkey.api.data.SqlExecutingSelector$ExecutingResultSetFactory.handleResultSet(SqlExecutingSelector.java:457) ~[api-23.7.0.jar:?]
	at org.labkey.api.data.TableSelector.forEachResults(TableSelector.java:273) ~[api-23.7.0.jar:?]
	at org.labkey.list.model.ListManager.indexEntireList(ListManager.java:839) ~[list-23.7.0.jar:?]
	at org.labkey.list.model.ListManager.lambda$deleteItemIndex$6(ListManager.java:592) ~[list-23.7.0.jar:?]
	at org.labkey.list.model.ListManager$$Lambda$1837/0x00007fc3d11bee58.run(Unknown Source) ~[?:?]
	at org.labkey.api.data.DbScope$CommitTaskOption.run(DbScope.java:2010) ~[api-23.7.0.jar:?]
	at org.labkey.api.data.DbScope$TransactionImpl.commit(DbScope.java:2327) ~[api-23.7.0.jar:?]
	at org.labkey.query.controllers.QueryController$BaseSaveRowsAction.executeJson(QueryController.java:4473) ~[query-23.7.0.jar:?]
	at org.labkey.query.controllers.QueryController$DeleteRowsAction.execute(QueryController.java:4622) ~[query-23.7.0.jar:?]
	at org.labkey.query.controllers.QueryController$DeleteRowsAction.execute(QueryController.java:4614) ~[query-23.7.0.jar:?]
	at org.labkey.api.action.BaseApiAction.handlePost(BaseApiAction.java:238) ~[api-23.7.0.jar:?]
	at org.labkey.api.action.BaseApiAction.handleRequest(BaseApiAction.java:128) ~[api-23.7.0.jar:?]
	at org.labkey.api.action.BaseViewAction.handleRequest(BaseViewAction.java:195) ~[api-23.7.0.jar:?]
	at org.labkey.api.action.SpringActionController.handleRequest(SpringActionController.java:512) ~[api-23.7.0.jar:?]
	at org.labkey.api.module.DefaultModule.dispatch(DefaultModule.java:1128) ~[api-23.7.0.jar:?]
	at org.labkey.api.view.ViewServlet._service(ViewServlet.java:240) ~[api-23.7.0.jar:?]
	at org.labkey.api.view.ViewServlet.service(ViewServlet.java:159) ~[api-23.7.0.jar:?]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) ~[servlet-api.jar:4.0.FR]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209) ~[catalina.jar:9.0.79]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[catalina.jar:9.0.79]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) ~[tomcat-websocket.jar:9.0.79]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[catalina.jar:9.0.79]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[catalina.jar:9.0.79]
	at org.labkey.api.data.TransactionFilter.doFilter(TransactionFilter.java:43) ~[api-23.7.0.jar:?]

I am running with 16GB of heap:

-Xms16G -Xmx16G -XX:-HeapDumpOnOutOfMemoryError

And the overall load on the system is quite low.

When investigating I looked at /admin-memTracker.view and was surprised to see ~30 instances of "SearchService:index". But I haven't looked at this page before, so perhaps that is normal.

Some other system details:
Servlet Container Apache Tomcat/9.0.79
Java Runtime Vendor Eclipse Adoptium
Java Runtime Name OpenJDK Runtime Environment
Java Runtime Version 17.0.8+7
Database Product Name PostgreSQL
Database Product Version 15.3

Is anyone else running into out of memory issues after moving to v23.7.0?

Thanks,
-Will