• Sign In
  1. Messages

RLabkey - Upload Taking a Long Time

LabKey Support Forum

View Message

view list print
RLabkey - Upload Taking a Long Time marcia hon  2018-02-09 05:28
Status: Closed
 
Hello,

We are trying to automate the process of uploading into Labkey.

We are trying to use RLabkey library to insert.

There are 50,000,000 records and we kept it running. Still after one week it is still uploading.

I tried uploading via the "Bulk Upload GUI" option, and it takes 25 minutes.

Please let us know how to get RLabkey to upload in a good time as we would like to automate this process.

Thanks,
Marcia
 
 
Jon (LabKey DevOps) responded:  2018-02-10 00:15
Hi Marcia,

Fifty million records is a lot of data to insert into any database in a single go, so your database and server should be setup to have a lot of RAM, CPU power, and memory to handle that.

If you are using the Rlabkey API, you should remember to use the labkey.importRows() function rather than the labkey.insertRows() function since you want the system to try and have it do a bulk import of the data rather than individual rows one at a time. See https://cran.r-project.org/web/packages/Rlabkey/Rlabkey.pdf for more details on the two functions.

Regards,

Jon
 
marcia hon responded:  2018-02-12 07:19
Thank you!