500 error trying to export a large query

LabKey Support Forum (Inactive)
500 error trying to export a large query michael_stover  2012-01-27 07:09
Status: Closed
 
I have a query that returns 35,000 rows of 12 columns, all numbers. The query takes a good minute to run (it is doing a pivot on about 1.3 million rows, still a minute seems excessively slow, and subsequent calls are never any faster).

When I try to export, either as an excel or as a tab file, I usually get a 500 error. In the logs, I see:

ERROR TextWriter 2012-01-27 10:07:47,921 TP-Processor5 : PrintWriter error

or:


ERROR ExcelWriter 2012-01-27 09:37:10,335 TP-Processor5 : ClientAbortException: java.net.SocketException: Broken pipe

An ideas? Anything I can do to improve the performance of these queries and exports?
 
 
Brian Connolly responded:  2012-01-27 07:51
Michael,
Can you provide us with little more information about your server and the query that you are attempting to run.

- What version of LabKey are you running
- What is the database type (postgresql or MS SQL Server) and the version that you are using?
- Information about your LabKey Server and database hardware.
-- Are they run on separate servers
-- # CPU, Memory
-- Is the hardware a desktop, server, laptop, etc
- What is the query you are attempting to execute and how are you running it?
- Can you send us an export from the Queries Admin Console
-- Goto your Admin Console -> click on Queries in the Diagnostics section and hit the export button at the top of the page.
- Can you send us the labkey.log file for your server. You can find that in the Tomcat logs directory.

Thank you,

Brian
 
michael_stover responded:  2012-01-27 09:45
Hi Brian,
  We are currently running Labkey 11.1.
We are running Postgres 9.x - I'm not entirely certain right now which.

The server is running both Labkey and Postgres together. specs:

12 cpu cores (athlon)
32GB memory, java heap starts at 4GB, can grow to 20
It's a poweredge server

I've attached the query export and the labkey log gotten via the web interface. In the query log, find "pivot". The first hit is the query in question.
 
Brian Connolly responded:  2012-01-27 09:49
Thank you for the information and the files. I will look at them now.

I have one last question, what is the configuration of your postgresql database like. I am particularly interested in the the settings
- shared_buffers
- wal_buffers
- effective_cache_size
- work_mem

You can find these in the file postgresql.conf


-Brian
 
michael_stover responded:  2012-01-27 10:01
Unfortunately I don't have access to that info. I've requested it and I'll get back to you.
 
michael_stover responded:  2012-01-27 12:36
see postgres conf attached
 
Brian Connolly responded:  2012-01-27 14:56
Michael,

The problems you are experiencing are most likely being caused by the configuration of your PostgreSQL server. From the configuration file that you sent me, it looks like the database is using the default configuration.

The default configuration for PostgreSQL is meant to work on servers with 512MB or memory or less. If you change the configuration of your PostgreSQL server, the performance of you server will likely improve.

Documentation on tuning your postgresql server is available at

http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server and https://www.packtpub.com/article/server-configuration-tuning-postgresql

If you do not want to take the time to review all the documentation above, changing the following settings should provide a significant performance improvement

-shared_buffers = 12% of total memory size(but no larger than 2GB)
-work_mem = 10MB
- maintenance_work_mem = 1024MB
- wal_buffers = 4MB
- checkpoint_segments = 10   
- checkpoint_timeout = 15min
- enable_material = off
random_page_cost = 1.5
- effective_cache_size = 75% of total memory size

NOTE: I know very little information about your server configuration and other programs running on the server, so I recommend consulting with your DBA/System Administrator before making these changes. Or using these values as a starting point and evaluating the server performance after making the change.

Lastly, I would recommend reducing the Max Heap setting in Tomcat to either 12 or 16GB as the PostgreSQL database will now be using more of the servers memory.

-Brian
 
michael_stover responded:  2012-01-30 05:30
Thanks for the help Brian!