• Sign In
  1. Messages

How do we ignore existing rows in a dataset when “importing bulk data” and import only new rows

General Server Forum (Inactive)

View Message

view list print
How do we ignore existing rows in a dataset when “importing bulk data” and import only new rows Chidi  2018-08-01 06:33
Status: Closed
 

Hi support team,

How do we ignore existing rows in a dataset when “importing bulk data” and import only new rows?

This feature exists when importing data in a sample set. Attached is a screenshot of the feature, and I highlighted the one that are important to us.

Is there a way we can include the feature to our datasets? This will be helpful because we can download an excel sheet, add new rows on our computer, and import the excel sheet successfully into labkey portal without getting a blank or an error message saying “some rows already exist in the dataset on Labkey”.

Thank you very much.
 
 
Jon (LabKey DevOps) responded:  2018-08-14 22:13
Hello,

Unfortunately, dataset bulk importing requires that the original file you're using for uploading be free of duplicate records.

https://www.labkey.org/Documentation/wiki-page.view?name=manuallyUploadStudyData

If the newer file you're uploading has all of the existing rows included along with new rows, you could delete all of the existing rows in the dataset and re-upload the newer file instead.

If you are interested in having this truncation of data done on the dataset and have your uploaded file replace the existing data, we do have a File Watcher feature that exists with our Premium versions of LabKey:

https://www.labkey.org/Documentation/wiki-page.view?name=fileWatcher

If the data is sitting on another database that LabKey can communicate with, you could also build your own ETL module to pull the data from your source table/query and send the data to the target dataset and both insert new rows and merge data with existing rows.

https://www.labkey.org/Documentation/wiki-page.view?name=etlModule

Please note that ETLs will be also apart of the Premium version of LabKey and will no longer be available in the Community edition of Labkey, starting in 18.3.

https://www.labkey.org/Documentation/wiki-page.view?name=changes182

Regards,

Jon