Hi,

We are planning to use Labkey Server to facilitate reporting needs for several of our researchers. We currently use an external service to collect and store our client's data. This service stores data in a relational database and comes with an API for accessing the stored data. I am currently trying to figure out the best way to automate Labkey integration with our data storage and would like some advice on what feature of Labkey would work best to facilitate automated integration. I can write a script to extract data from the external service fairly easily. The API is available for most major programming languages, so choice of scripting language is not that important (with the notable exception of SQL).

What I would ideally want to be able to do is use an existing Labkey feature to schedule the running of the script and put the output into a Labkey dataset. I've narrowed it down to a few Labkey features that might be able to help, but am not sure which of these is best suited to my task:

 1) ETL module: The ETL module seems to capture the essence of what I want to achieve except that the "Extract" feature appears to be limited to SQL queries and I would need to invoke a script in a different language (R or python or Java etc.).
 2) Develop a custom module: I am aware that Labkey allows developers to make custom modules for particular purposes. This seems powerful, however I feel like I would be reinventing the wheel if I went this approach and would like to know what exists already that I am missing.
 3) Labkey pipelines: This is a way to run a series of scripts. Does anyone know if these can be used to load data from external sources on a particular schedule?
 4) External Schemas (https://www.labkey.org/home/Documentation/wiki-page.view?name=externalSchemas): This appears to require direct access to the relational database that our external service uses to store the data. It might be possible for me to expose those tables to Labkey, although for reasons that should be obvious I would like to avoid doing that.

Finally, it is in my capability to set up a service independent of Labkey that sends data to Labkey via its API. I would prefer to use a Labkey feature if possible (mostly to keep Labkey related code internal to the Labkey server) but will do it this way if there is nothing within Labkey that fits my use case.

Thank you and feel free to reply with some questions if anything I wrote is unclear.