Indexing of large lists

General Server Forum (Inactive)
Indexing of large lists phains  2018-11-19 20:44
Status: Closed
 

I have an installation of LK Community Edition running on Win server 2012.

I am uploading some very large lists (up to 1.7 million lines). Part of my goal in uploading these lists is that I can search them all at the same time for their contents using the LK “search” function. The problem is, not all the lists seem to be getting indexed, what I assume takes place, to allow for rapid searching.

When I first loaded the lists, searching returned no results. Leaving them over the weekend, most of the smaller lists now return matches with the same search terms that failed previously. Hence my theory about the lists getting indexed in some way. These smaller lists only contain ~200,000-400,000 lines.

So far, after about a week, the larger lists still do not return any matches, even when I know they contain the search term. I can filter each list individually using the search term directly in that column of the list, it works fine and is quite fast.

Are there any limits on list size and what will work with the generic LK search function? Is there something I can do to get this working on my system with the larger lists?

Thanks for any suggestions,

Peter

 
 
phains responded:  2018-11-21 19:38

I should have said it's LK 18.2. I've since uploaded some more smaller lists and these are searchable as expected. I'm leaning towards a size limit. Can anyone comment on this?

 
Jon (LabKey DevOps) responded:  2018-12-05 20:34
Hello Peter,

All objects should be indexed within the system unless if you're purposefully omitting them as described in our docs:

https://www.labkey.org/Documentation/wiki-page.view?name=searchAdmin

With lists specifically, you can dictate how the indexing is done:

https://www.labkey.org/Documentation/wiki-page.view?name=editListDesign#index

There could possibly be an issue with the size and there are also potentially memory issues when it comes to search as well, so you'll need to make sure your heap memory is large enough to satisfy a larger indexed file size limit.

https://www.labkey.org/Documentation/wiki-page.view?name=configwebappmemory

Regards,

Jon
 
phains responded:  2018-12-06 12:28
Thanks Jon,

a combination of changes seems to be working for me. I'm still working on it, but I can now see my lists that were not previously indexed, are getting indexed. I’d given up hope of working this one out.

Peter