Response Processing

2022-08-17

Submitted data, in the form of JSON objects, is stored in the Response table. The Response table holds the raw JSON objects before they are parsed into the individual data tables. The Response table also forms a dashboard for viewing errors associated with parsing and for manually kicking off reprocessing of responses.

Parsing and Processing

The Responses table keeps track of whether responses were parsed successfully or not. Unsuccessfully parsed responses can be re-processed. Common errors in processing include:

  • A question’s answer is not the expected type.
  • A table name doesn’t match any List in the schema.
  • A column in the response does not exist in the expected List.
  • The format of the response JSON does not match the expected format.
To view the Responses table:
  • Go to (Admin) > Developer Links > Schema Browser, open the schema mobileappstudy, and click Response, then View Data.
    • OR
  • To create a handy link on your study page, add a Query web part to display the mobileappstudy schema. For details see Set Up a Test Environment.

Reprocessing

From the Response table, editors can see the status of an event and can reprocess only responses that have not previously been successfully processed. Reprocessing is useful if the errors were caused by bugs in the processing code that have since been fixed. To reprocess, select the checkbox of a record(s) and click Reprocess.

Editing

To edit, hover over a row to expose the (pencil) icon for editing. There is no ability to edit the responses before reprocessing.

Processed Data and Dynamic List Creation

Successfully processed data is stored in simple tables called "Lists". The names of the tables and their columns are determined by the shape of the survey document submitted.

There is a List for each activity (e.g., InitialSurvey, WeeklySurvey, MonthlySurvey, ActiveTask1, etc.). The activity List contains the participantId and a column for each single-valued response in the survey. These are responses whose type is one of the following:

  • scale
  • continuousScale
  • textScale
  • valuePicker
  • boolean
  • numeric
  • timeOfDay
  • date
  • text
  • email
  • timeInterval
  • height
  • location
For these responses, the column name is the same as the key value for the question.

For example, the response JSON submitted under the activity "InitialSurvey":

"results" : [
{
"resultType" : "date",
"key" : "dueDate",
"value" : "2017-10-17"
}
]

results in a List named "InitialSurvey" with a field "DueDate":

InitialSurvey
DueDate

For each multi-valued response (i.e., those of type imageChoice, textChoice, or grouped), there is a separate List with a name that follows the pattern <survey name><key>.

For example, the following response JSON:

"results" : [
{
"resultType" : "textchoice",
"key" : "supplements",
"skipped" : false,
"value" : [
"Q10",
"B12"
]
}
]

results in the following tables and fields:

InitialSurvey
InitialSurveySupplements
Supplements

For grouped results in which each response in the group is single-valued, there is a new List created that corresponds to the grouped response’s key and with a column in the table for each response within the group. For imageChoice and textChoice fields, there is a List for the response with the column name corresponding to the response key.

For grouped results in which there are responses that are multi-valued, there is a List created that corresponds to the grouped response’s key value with a column in the table for each single-valued response in the group. There is also a separate List created for each multi-valued response in the group. The name of the List is <survey name><group response key><response key> and it contains a foreign key to the grouped result List.

For example, the following response JSON:

"results" : [
{
"resultType" : "grouped",
"key" : "rx",
"value" : [
{
"resultType" : "textchoice",
"key" : "medName",
"value" : [
"Acetaminophen"
]
}
]
}
]

results in the following Lists and fields

InitialSurvey
InitialSurveyRx
InitialSurveyRxMedName
MedName

The following three active task types are supported as special grouped results for which certain specific fields will be created in their corresponding Lists.

resultTypeFields
fetalKickCountercount (Integer), duration (Integer)
towerOfHanoipuzzleWasSolved (Boolean), numberOfMoves (Integer)
spatialSpanMemoryscore (Integer), numberOfGames (Integer), numberOfFailures (Integer)

When a survey design contains one of the resultTypes shown above, a List will be created just as with a grouped result containing the fields indicated.

For grouped results within grouped results, we’ll generate Lists recursively according to the above pattern.

The List structure is denormalized so all Lists contain the ParticipantId column, which makes it easier to perform filtered queries on any of the Lists. (This is the rowId from the participant table in the mobileAppStudy schema.)

Lists natively support many data analysis and reporting features, including

  • Filtering survey results by a particular participant to track changes to that participant’s responses over time.
  • Unioning multiple Lists based on a participant id to view all of that participant’s responses from multiple surveys.
  • Creating plots and visualizations.
  • Exporting all or filtered results to Excel, text, or other common formats.
  • Using the SAS API, Rlabkey API, or other language bindings to extract responses into SAS, R, or another tool/language.

Response Forwarding

An administrator can configure the Response Server to forward responses to another server. For instance, copying processed responses to a remote Salesforce server might be required in a particular kind of study.

Folder administrators can enable and disable response forwarding for a given folder with two types of authorization:

  • Basic Authorization: Provide a username and password, plus the endpoint URL.
  • OAuth: Provide a token request URL, token field, header name, and endpoint URL.
  • Disabled: (Default) Disable forwarding.
  • Select (Admin) > Folder > Management.
  • Click the Response Forwarding tab.
  • By default, forwarding is disabled. To enable, choose one of the options and enter the required information.
  • Click Submit.

Once response forwarding is enabled, a pipeline job is enabled to post the responses to the configured external server. This job will run every 5 minutes as well as after every successful response processing, as long as the job is 'succeeding'. The forwarding status is recorded in the Responses table.

  • Processed: A response that has been processed but not yet successfully forwarded.
  • Forwarded: A response that has been successfully forwarded.
  • Error: An error was seen at the last processing attempt.
The pipeline job queries the Response table for every row whose status is "PROCESSED", but not yet "FORWARDED". For each such row, post the response JSON and the corresponding enrollment token to the configured URL, authorizing the request by conveying configured credentials via basic authentication; wait for the external system to respond:
  • If the external system responds with "success", the status of the corresponding row in the Response table is updated to "FORWARDED" and this response will be never sent again.
  • If any failure occurs (can't connect, timeout, failure response), the job exits immediately and the Response server will wait and re-try that post at the next 5 minute interval.
If a full batch of responses is sent successfully, the external system will be flagged as "succeeding", otherwise, it will be flagged as "failing."

Responses that have been already forwarded successfully will not be reforwarded.

Related Topics