LSID generation for Study datasets

LabKey Support Forum (Inactive)
LSID generation for Study datasets Anthony Corbett  2012-05-14 08:03
Status: Closed
 
Labkey server version 12.1

I'm trying to setup a study dataset, called Progress Notes, which will allows N number of entries, ie. rows, for any participant and is not tied to a visit (Sequence Number). In order to achieve this I performed the following steps when creating the dataset:

1. Marked the dataset as a 'demographic' dataset. I'm assuming, please correct me if I'm wrong, that a demographic dataset means that the entries are associated with the PariticpantID regardless of Sequence Number, and thus associated to the Participant across the 'whole' of the study.

2. In order to make a row unique (as there is no Sequence Number) I added an integer field, named noteID, which I set to be managed by Labkey so that it will auto-increment the value making the composite ParticipantID-noteID key unique per row.


Unfortunately, the managed noteID is not used in the generation of the lsid (the real PK). I get the following error when trying to insert a new record for a participant that already has a row in the dataset:

ERROR: duplicate key value violates unique constraint "c14d117_progress_notes_pk"
  Detail: Key (lsid)=(urn:lsid:urmc.rochester.edu:Study.Data-14:5015.01-13) already exists.



If I make the dataset non-demographic the lsid will generate with the managed (auto-incremented) id at the end.

    urn:lsid:urmc.rochester.edu:Study.Data-14:5015.01-13.11.0000.12

    urn:lsid:urmc.rochester.edu:Study.Data-14:5015.01-13.11.0000.10

However, this forces a sequence number (11 in the lsid's above) to be specified which isn't applicable to the progress notes, which can be added at any time in my use case.


I prefer not to use lists for the following reasons:

1. There is no convenience of automatic inserts of created/createdBy and modified/modifiedBy for lists. These fields will have to be additional fields on the list which will cause a lot of boiler plate code to be written on the client side business logic before insertion. Really I can't trust all the client side data as authoritative and the server side should really be doing this.
 
2. The Entity ID guid does not show up in the audit log for list events so there is no way, that is clearly evident, to produce an audit trail (Even though a lsid changes over time [per modification] making this difficult anyways).

3. Even though I can set the ParticipantID field in a list to the 'Subject/Participant (String)' look-up type I loose the hyperlink to the Participant overview from the ParticipantID column in the 'view data' grid. In addition Labkey's EXT4 editableGrid doesn't see that this list field is a Subject/Participant lookup and therefore I loose the dropdown of possible ParticipantID to choose from, this making the user entry more error prone. The editable grid populates fine with possible ParticipantIDs when the source is a study dataset. To me this points to a metadata issue with lists that have a lookup type of Subject/Participant.



Is it possible to generate the lsid using the managed ID field with a demographic data set?


If this isn't clear or you need more information please let me know.


Thanks,

Anthony Corbett
University of Rochester
Dept Biostatistics and Computational Biology
 
 
Matthew Bellew responded:  2012-05-14 13:47
Currently this is not possible. For visit-based studies you have these options for the unique dataset keys

  PTID (demographic)
  PTID, SequenceNum (clinical data often)
  PTID, SequenceNum, ExtraKey (assay data often)

If the SequenceNum field is not important, I would suggest creating a dummy visit called "Progress Notes", and use that visit for all the rows in that dataset. It would be nice if we provided for auto-filling that value, but we don't have that feature yet.

Matt
 
Anthony Corbett responded:  2012-05-18 14:04

Thanks Matt for the response. I now have a dummy visit (Sequence Number 101) setup just to hold progress notes with an additional auto-incremented key. I've take the visit off the overview so it looks like it doesn't exist in the study navigator.

As a follow up to this I now have additional questions.

I'm looking to add an LABKEY.ext.EditorGridPanel for progress notes to the Participant Overview (filtered by the current Participant being viewed in the overview). The javascript looks like this:



    /* get the participant id from the request URL: this parameter is required. */
    var participantId = LABKEY.ActionURL.getParameter('participantId');

    var _grid;

    //Use the Ext.onReady() function to define what code should
    //be executed once the page is fully loaded.
    //you must use this if you supply a renderTo config property
    Ext.onReady(function(){
        _grid = new LABKEY.ext.EditorGridPanel({
            store: new LABKEY.ext.Store({
                schemaName: 'study',
                queryName: 'progress_notes',
                columns: 'ParticipantId,SequenceNum,date,noteType,CreatedBy/Email,notes',
                filterArray: [    LABKEY.Filter.create('ParticipantId', participantId) ]
            }),
            renderTo: 'grid',
            width: 1000,
            autoHeight: true,
            title: 'Progress Notes',
            editable: true
        });
    });

My issue comes in when I try to customize what columns are showing and adding a new row when 'required' fields aren't showing.

I'd like to hide the ParticipantId and SequenceNum columns, the but following two solutions don't work.

1. If I don't specify these columns when creating the store they won't be there. Howeevr if I then try adding a new row I get an error when it goes to the server to save that it needs both of those fields.

2. I specify the ParticipantId and SequenceNum columns in the store config but I remove them from the grid. The behavior is that validation fails as those are required fields (even though I can't see them) so it never tries to save the row and doesn't even give user feedback that validation failed. This seems correct as in the EditorGridPanel.js file the following is documented:

    /**
     * Saves all pending changes to the database. Note that if
     * the required fields for a given record does not have values,
     * that record will not be saved and will remain dirty until
     * values are supplied for all required fields.
     * @name saveChanges
     * @function
     * @memberOf LABKEY.ext.EditorGridPanel#
     */
    saveChanges : function() {
      ...
    }



I think I can work around this if I can hook in a listener on the store so when a new row is created or added I can autofill certain fields. I'd like to be able to autofill the following columns:
ParticipantId (we already know that from the query param),
Sequence Num should always be 101 (my dummy visit),
Date with current date,
Name is the email of the person creating the record (doesn't matter what I put in this column actually as it will be overriden by the server with the createdBy user)


So, my two questions are:

1. How do I specify custom configuration to always hide the ParticipantId and SequenceNum columns? Is this columnModel metadata I can specify inline to override what is coming back from the query?
2. How can I get a listener on the store to auto fill row fields for new records?


Thanks a ton!

-
Anthony
 
trent responded:  2012-05-20 19:11
Hi Anthony,

1. How do I specify custom configuration to always hide the ParticipantId and SequenceNum columns? Is this columnModel metadata I can specify inline to override what is coming back from the query?

Try setting the config property, on the grid, 'autoSave' to false. It requires the user to hit the save changes button before saving, but aside from it not throwing the aforementioned error, its more user friendly imo.

2. How can I get a listener on the store to auto fill row fields for new records?

Not sure if this is the best way or not, but I tied a beforecommit event to the store, for my use.

store.on('beforecommit', function(records, rows){
    for(i=0;i<rows.length;i++){
        rows[i].values["patient_visit_id"] = patientVisitId;
    }
    }
);

rows[i].values is a JSON object that represents each column/value that is sent to the server to commit to the table.
 
Ben Bimber responded:  2012-05-21 07:43
Hi Anthony,

for #1, you might want to check out the columnmodelcustomize event on EditorGridPanel. this is called when the server returns w/ column metadata. it provides an array of columns, I believe. you can iterate that array, find the columns you want to hide and set 'hidden: true' on them. you can do a similar trick with the store using the 'metachange' event. in general it's often better to set metadata at this lower level, for example, if you plan to share this store between components; however, you're almost certainly ok using columnmodelchange for this case.

for #2, i might suggest listening for the store's update or add event. on update or add, you can modify the record and set the value for whichever fields you need. when doing this, be careful you first suspendEvents(), then resumeEvents() on the store before setting those fields or you may get yourself into a never ending loop.
 
Anthony Corbett responded:  2012-05-21 13:55

Thanks Trent and Ben for the tips. I got this working on with your suggestions. In the end it modifies the columnModel like I wanted to be able to to hide the two columns where still auto populating them when a new record is added. See code below for reference:

<script type="text/javascript">
    /* get the participant id from the request URL: this parameter is required. */
    var participantId = LABKEY.ActionURL.getParameter('participantId');

    var _grid;

    function customizeColumnModel(colModel, index, c) {
        //Hide columns
        index.ParticipantId.hidden = true;
        index.SequenceNum.hidden = true;
        index.date.renderer = Ext.util.Format.dateRenderer('m\\/d\\/Y');
    }

    function onAddProgressNote(store, records) {
        store.suspendEvents();
        for (var i=0 ; i<records.length ; i++) {
            records[i].set('ParticipantId', participantId);
            records[i].set('SequenceNum', 101);
            records[i].set('date', new Date().format("Y/m/d") );
            records[i].set('CreatedBy/Email', LABKEY.Security.currentUser.email);
        }
        store.resumeEvents();
    }

    //Use the Ext.onReady() function to define what code should
    //be executed once the page is fully loaded.
    //you must use this if you supply a renderTo config property
    Ext.onReady(function(){
        _grid = new LABKEY.ext.EditorGridPanel({
            store: new LABKEY.ext.Store({
                schemaName: 'study',
                queryName: 'progress_notes',
                columns: 'ParticipantId,SequenceNum,date,CreatedBy/Email,noteType,notes',
                filterArray: [    LABKEY.Filter.create('ParticipantId', participantId) ],
                sort: '-date',
                listeners: {
                  add: onAddProgressNote
                }
            }),
            renderTo: 'grid',
            listeners: {
              columnmodelcustomize: customizeColumnModel
            },
            width: 1000,
            autoHeight: false,
            height: 300,
            title: 'Progress Notes',
            editable: true,
            autoSave: false,
            enableFilters: true,
            enableColumnHide: false,
            firstEditableColumn: 5 //Start editing on the noteType column
        });
    });
</script>

Thanks so much!