Problem with copying Expression Matrix assay run data to a study

General Server Forum (Inactive)
Problem with copying Expression Matrix assay run data to a study jytian  2018-11-20 18:52
Status: Active
 

Hi LabKey Server support: Thank you for your help in advance.

I have an "Expression Matrix" assay, and I have successfully imported the "sample information" file, "feature annotation" file and "gene expression matrix" file into the assay. The assay run result is a table of 4 columns: "Value", "Probe Id", "Sample Id" and "Run". I want to copy this table to a Study. When doing so, LabKey asks me to provide "Participant ID" and "Visit ID" for each row. My problem is: I have tens of thousands of rows and there is no way I can manually enter these values for all rows. There must be a way to get these two fields automatically mapped, but I have tried all I could and read LabKey documents I could find and I cannot find a way of doing so. The LabKey tutorials on this subject apply to many other assay types but not to "Expression Matrix" assay type. I tried to add a "ParticipantVisitResolver" field in the assay's design (the batch property) and LabKey doesn't accept it. I customized the grid view of the run data (the table I want to copy to the study) by adding the "Participant ID" and "Visit ID" columns to it (they are linked to the sample id in the sample set so I can add them to the table), but LabKey doesn't recognize them and still asks me to provide values for those two fields. Please give me a clue on how to automatically map these two fields. Your help is greatly appreciated!

Regards
Joann

 
 
jytian responded:  2018-11-21 19:07

Adding:

Following the LabKey's document "Using Sample Sets as an Alternative to s Specimen Repository", I created a List mapping the sampleId to the ParticipantId and VisitId. Then I added a field "participantVisitResolver" in the assay design's batch fields, and then tried to point this resolver field to the List using Look Up field type. As you can see from the attachment image, the LabKey lists my list as option but I cannot select the list because the list is not selectable. I tried to manually text the list name to the location but it doesn't work (somehow LabKey doesn't recognize the List). I have a feeling that Expression Matrix assay type is different from many other assay types, because the run data result is a table whose columns are created by the LabKey Server, and the columns don't exist in the file (to be imported). I am not sure we are suppose to edit the batch fields.

Linking the assay design to the Sample Set (as indicated in the above-mentioned document) doesn't work either, because sampleId is not a field in the assay's design.

Please advice. Thanks a lot!
Joann

 
Jon (LabKey DevOps) responded:  2018-12-05 21:18
Status: Closed
Hi Joann,

What version of LabKey are you working with?

How is the ParticipantVisitResolver not being accepted exactly?

The feature is pretty straightforward (https://www.labkey.org/Documentation/wiki-page.view?name=participantVisitResolver) and usually will map over.

Can you possibly provide us a small demo of your assay and some mock data you're working with so we can try to look into the issue? You should be able to create a XAR archive of your existing assay (https://www.labkey.org/Documentation/wiki-page.view?name=experiment) or you can export the folder/project container of your demo assay along with your list that contains the participant/visit resolver details.

Regards,

Jon
 
jytian responded:  2018-12-06 19:00
Status: Active

Hi Jon:

Thank you for your response! My answer is long so I will divide it to two posts. First to answer your questions:

  1. The version of the LabKey Server I am working with is: 18.2.

  2. You said: “The feature is pretty straightforward (https://www.labkey.org/Documentation/wiki-page.view?name=participantVisitResolver) and usually will map over”. This is exactly my problem. I am sure that the documentation provided in this link is straightforward for many other assay types, but it doesn’t work with the "Expression Matrix" assay type for me:

a) For "Expression Matrix" assay type, the “Batch Fields” section in the assay designer page doesn’t look the way as the documentation in the link indicated. It looks like in the attached "image1" file. There is no default batch field for this assay type, and there is no where one can specify the “Sample indices” mapping as the documentation in the link indicated. The “Using Default Values” part of the documentation doesn’t apply this "Expression Matrix" assay type either.

b) I tried to manually add a “ParticipantVisitResolver” field in the “Batch Fields” area (using the “Add Field” button), and choose the field type as “Lookup”. Since the "Sample Set" doesn't help as I think it should, I created a "List" to map the samples to their participant ids and visit ids using attached files “mapping_list” or “mapping_list_index”, all possible ways I could. The “Lookup” does see the Lists I have created but it cannot select anyone of them because none of the Lists is select-able (see attached "image2" file).

c) I think that the “Sample Set” in the "Expression Matrix" assay type should do the mapping, but it doesn’t work that way.

I will provide my demo assay and data in next post.

Thanks for your patience.
Joann

 
jytian responded:  2018-12-06 19:37

(continue)
Hi Jon:

Hereby I provide you a small demo of my study and assay with my data so that you can look into the issue. Please do the following steps with the
files I attach to see the issue:

  1. Create a visit-based Study. Import the attached “Demographics” file to a “Demographics” dataset. Edit the Dataset Definition to make sure that it is “Demographic Data”. Import the attached “PhysicalExam” file to a “PhysicalExam” dataset. You can then manage the “Visit” or not, I don’t think it matters.

  2. Create an "Expression Matrix" assay by doing the following steps (according to LabKey's Expression Matrix Assay Tutorial at https://www.labkey.org/Documentation/wiki-page.view?name=geoMicroarrayTutorial):

a). Create a subfolder of Microarray type (I don’t think the location matters but I created it in the location there the Study is located, namely parallel with the Study).

b). Add a “Sample Sets” web part in this assay page. Import a Sample Set by using the attached “sample_info” file.

c). Add a “Feature Annotation Sets” web part in this assay page. Import a Feature Annotation Set by using the attached “feature_annotation” file.

d). Please note that my assay run data is in the attached “run_1” file.

e). In the “Assay List”, create a new assay design of “Expression Matrix” type (assay location is the current folder). Now in the Assay Designer page you see that the “Batch Fields” section has nothing as I said in my previous post. You can accept the default settings and then import assay run data using the attached “run_1” file (specify the Feature Annotation Set in the process).

f). Open the imported run data. Select some rows or all rows to “Copy to Study” (to the Study we created). You will see that the data are not automatically mapped to their Participant IDs and Visit IDs.

g). Now edit the assay design by defining the “ParticipantVisitResolver” field in the "Batch Fields" section (then re-import the run data of course). As mentioned in my previous post, I have tried all the possible ways I could, creating the Lists, trying to use the "Lookup" field type as I described in my previous post, but nothing works for me.

I have admin permission on all of my folders.

My goal is to get the imported assay run data automatically mapped to their Participant IDs and Visit IDs when copying the data to the study. Please advise how I can get it work.

Thanks a lot!
Joann

 
Jon (LabKey DevOps) responded:  2018-12-24 14:16
Status: Closed
Hi Joann,

Thank you for the detailed repro steps. I was able to confirm the behavior you described and it does appear to be specific to Microarray.

It is possible that this functionality was not incorporated within Microarray, but I'll get this confirmed.

Thank you for your patience.

Regards,

Jon