Yes, as you have seen there is now a limit on the number of columns that can be created for a list. This change first occurred in 14.1 where we changed the data representation of lists to hard tables in the database. Unfortunately, this has the side effect of limiting the number of columns in a list to the number of fields supported in a database table. When the limit is exceeded, the somewhat ungraceful postgres error is generated. My advice would be to normalize the list into multiple lists with fewer columns that could be joined together in a query on a key value.

Hope this helps,
Greg

An additional message from Alicia:

Hi Greg,
 
Thank you for your response.
 
I am not working with a list. I am working with an existing dataset with the established columns. Many of the columns are assigned to lists.
 
I am working on breaking the dataset into two different datasets. This is becoming an arduous task because every subject has MV values within the columns throughout the table.
 
I have to export all 64 columns (and ensure only the key values are obtained from the lookups) and in addition, I have to customize the export to include all of the MV values (if any) for each column.
 
Subsequently, I have to then write a script/program to ensure the data is the same and then parse the records back into the two new datasets.
 
Is there an easier to way export data from a dataset with the raw values, and if applicable, the MV value.
 
Thanks and I look forward to your response,
 
Alicia Tyrell

Hi Alicia,

Given that you are working with a dataset, this does not appear to be related to the change of lists to hard-tables. I will investigate this issue again and see if I can come up with additional information about the cause of this error. We know that there is a limit on the number of parameters to a postgres function but I need to pinpoint exactly which function is being called. Did you happen to set a lot of MV indicators before this problem was observed? I would recommend waiting on splitting the dataset into separate tables until after we know more.

Greg

Hi Alicia,

We've opened an issue for this problem:

https://www.labkey.org/issues/home/Developer/issues/details.view?issueId=21234

I believe that this issue exists for both 14.1 and 14.2. For the time being, I've assigned it to 14.1, though we'll need to verify that the proposed fix will also work there.

Thanks,
Josh

Hi Greg & Josh,

Acknowledged.

Also, is there a way to address the problem of the MV value being excluded from the export of a dataset, in which the columns have the MVIndicator option enabled, the MVs are never included in the export results.

You essentially have to customize a view to obtain the MVs (and then write scripts to merge the data), or via the schema browser, write case statements to grab either the column or its MV and then export it.

I look forward to your response.

Alicia

Hi Alicia,

The original problem of the import feature being broken for a dataset with a large number of fields has been fixed in LabKey versions 14.1 and 14.2. We have sent a LabKey 14.1 installer to your LabKey Server Builds page on your support portal. Please install the build and test on your development server and let us know if you have questions about the install process. Also note that the fix involves disabling import 'merge' into a dataset; insert is supported instead. What this means is that existing key (subject IDs / Date) values cannot be in the excel file being imported because this will cause a conflict and produce an error. Insert only works if those dataset rows have first been deleted. If you have questions or concerns about this behavior, let us know and we can discuss them.

LabKey Server Builds Page:
https://www.labkey.org/project/Rochester/LabKey%20Server%20Builds/begin.view?

Regarding your question on MVIndicator being exported, we will need to investigate further before responding.

Greg

Hi Alicia,

This is the current expected behavior on export. I don't have any other suggestions beyond those that you already mentioned in terms of how to include both the "real" value and the missing value indicator in the output. We made the decision to include just the "real" value in the export case because it caused problems for many downstream tools when there were mixed types in a single column. For example, having integer or date values for many rows, but a string missing value indicator in others.

In your usage scenario, are "real" values and missing value indicators mutually exclusive, or do some rows have both?

Thanks,
Josh