Example Code: Split Large Sample Upload: /Documentation

Example Code: Split Large Sample Upload

When a sample upload is very large, it can be advantageous to split it into smaller "chunks" to avoid timeouts and other errors. The example on this page shows a python script that will split a large csv file into batches of 1000 samples, avoiding such errors.

The function bulkUploadSampleFromCSV takes a file and a boolean "should_update" argument:

bulkUploadSampleFromCSV(FILE_NAME, True) processes through a file in update mode. Both existing and new samples may be included in the file.
bulkUploadSampleFromCSV(FILE_NAME, False) processes through a file in insert-only mode (non-update). The upload will fail if any rows are included for duplicate sample IDs. This is also the default if no second argument is provided.

Be sure to edit the top portion of our example script to substitute your own values for the server, container path, API key, sample type, file name, etc. The context path will be either 'labkey' or an empty string '' as shown below.

Example Code

Download a file containing this script: bulkUploadScript.txt

from __future__ import unicode_literals
from labkey.api_wrapper import APIWrapper
import csv

# Before running this script, edit to set the required values for these variables, including your APIKey

LABKEY_SERVER = 'myserver.domain.org'
CONTAINER_PATH = 'project/folder/subfolder'
CONTEXT_PATH = ''
USE_SSL = False
API_KEY = 'PASTE_API_KEY_HERE'
api = APIWrapper(LABKEY_SERVER, CONTAINER_PATH, CONTEXT_PATH, use_ssl=USE_SSL, api_key=API_KEY)
SAMPLE_TYPE_NAME = 'sampleTypeName'
FILE_NAME = 'data.csv'


def upload_samples(server_context, container_path, sample_type, csv_payload, start_index, end_index, should_update):
    import_samples_url = server_context.build_url('experiment', 'importSamples', container_path)
    import_samples_url += '.view?query.queryName={0}&schemaName=exp.materials'.format(sample_type)
    request_payload = {
        'format': 'csv',
        'text': csv_payload
    }
    if should_update:
        request_payload['insertOption'] = 'MERGE'

    response = server_context.make_request(import_samples_url, payload=request_payload, method="POST")
    if response['success']:
        print("Success: Processed rows {0}-{1}".format(start_index, end_index))
    else:
        print("Error: {0}. Did not process rows {1}-{2}".format(response['exception'], start_index, end_index))


def bulk_upload_sample_from_csv(filename, should_update):
    with open(filename, 'r') as csvFile:
        csv_reader = csv.reader(csvFile)
        column_headers = ','.join(next(csv_reader))

        csv_payload = column_headers
        current_index = start_index = end_index = 0
        for row in csv_reader:
            if current_index >= 1000:
                upload_samples(api.server_context, CONTAINER_PATH, SAMPLE_TYPE_NAME, csv_payload, start_index, end_index, should_update)
                csv_payload = column_headers + "n" + ','.join(row)
                start_index = end_index
                current_index = 1
            else:
                csv_payload += "n" + ','.join(row)
                current_index += 1
            end_index += 1
        upload_samples(api.server_context, CONTAINER_PATH, SAMPLE_TYPE_NAME, csv_payload, start_index, end_index, should_update)
    print("Processing {0} has finished".format(FILE_NAME))


# bulk_upload_sample_from_csv(FILE_NAME, True)

LabKey Support

LabKey Support