Hello,

Your list looks pretty complete to me in terms of possible approaches. Some additional context that may be helpful:

1. ETLs are run as LabKey pipelines under the covers, and you can inject custom tasks, implemented in a custom Java module, into their execution sequence. If you have the LabKey Server source code, you can find a test/example at:

server\test\modules\ETLtest\resources\ETLs\appendAndTaskRefTask.xml

It uses the custom task as implemented here:

server\modules\dataintegration\src\org\labkey\di\steps\TestTaskRefTask.java

However, you're correct that there is not currently a way to pipe in data from a non-SQL source. ETLs have a convenient feature of being able to run on a schedule.

3. You could certainly use a script or custom Java task that pulls from an external source. The pipeline module itself does not currently have a scheduling mechanism though.

4. Yes, this would be possible, but would of course avoid the API that you're looking to leverage for extracting the data.

The server does have the Quartz library behind the scenes, which can be used for cron-like scheduling in custom modules. It's what the ETL system uses underneath to kick off its jobs on the requested schedule.

Thanks,
Josh