database size Ben Bimber  2010-08-26 10:41
Status: Closed
 
a little while back we had an issue where our DB was bloated w/ a lot of extra records in the audit table. prior to clearing these records I was unable to successfully restore a pg_dump of our production server into my development machine. after clearing these, the size went down and i was able to restore.

over the past 2 weeks we've moved a substantial amount of data in and out of labkey. we reloaded the whole EHR study. in theory this should be a relatively neutral move, since the rows deleted roughly equals the rows inserted. the filesize of our nightly postgres dumps has gone up considerably compared to before. i am once again getting errors when trying to restore that pg_dump file on my machine.

would it be possible to send any of the queries or other troubleshooting you guys did to identify problems last time? i have no reason to suspect that audit is the problem; however, something is much bigger than before.
 
 
Brian Connolly responded:  2010-08-26 16:34
Ben,
The previous problem that you encountered was due to the fact that when you performed a backup, you were specifying the output format to be "tar". When using this format, no single table can be larger than 8GB in size. (At the time, the audit table in your database was larger than 8GB in size)

To determine if the backup of any given table is larger than 8GB in size, all you need to do is untar the backup file in a temporary directory. This will produce a file for each table in your database. If any of the files are larger than 8GB, then you will not be able to restore that table.

If you are still using the "tar" output format, then my recommendation is to use the "custom" output format. This format does not have the 8GB limitation. To use this format, simply add the option, "--format=c" to you pg_dump command.


If you are not using the "tar" output format, then can you send me the error message you are receiving when you attempt to restore and I can try and help you.



Other Commands that might be of interest:
================================================

1) Determine the size of your database on disk

SELECT pg_size_pretty(pg_database_size('DBNAME'));


2) Determine size of a given table on disk (this includes both the table and all indexes)

SELECT pg_size_pretty(pg_total_relation_size('big_table'));


3) List row count for all tables in your database

SELECT n.nspname as "Schema", c.relname as "Name", c.reltuples as "Rows" FROM pg_catalog.pg_class c LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
WHERE c.relkind = 'r' ORDER BY c.reltuples DESC