Joining external and internal schemas

LabKey Support Forum (Inactive)
Joining external and internal schemas eva pujadas  2016-08-12 08:50
Status: Closed
 
Dear LabKey developers,

We would like to join tables from and external schema with tables from the internal schema on LabKey Server. Since it is not possible to do that through the web interface, which would be the recommended way to join such tables?

Is it possible to query external schemas using some API (e.g. Python)?

Our external database is MySQL, and the internal LabKey one is Postgres. The MySQL table we would like to join contains about 1 million rows.

Thank you for your help.
Regards,
Eva
 
 
Jon (LabKey DevOps) responded:  2016-08-12 10:31
Hi Eva,

Although you technically could use a scripting language like python to do this, this would require some significant scripting work depending on the type of join you're trying to do between the two tables. Most of the examples I've seen of people doing this required them to pull down the data into csv files, then have the csv files used in conjunction with something like pandas (http://pandas.pydata.org/pandas-docs/stable/index.html) to do the merging/joining (http://pandas.pydata.org/pandas-docs/stable/merging.html). This information would also be sitting outside of LabKey rather than within the UI, so you wouldn't be able to do anything beyond just looking at the joined data via Python.

Have you considered possibly importing that external table into your LabKey database through an ETL? You could setup a whole unique internal schema in LabKey and a table for this MySQL data that could get updated regularly and thereby allowing you to do everything you want to do within the LabKey UI.

https://www.labkey.org/home/Documentation/wiki-page.view?name=etlTutorial

Regards,

Jon
 
adam responded:  2016-08-13 07:48
As Jon has indicated, LabKey doesn't support direct joining of data from different database servers; we've tried to make that clear in the documentation: https://www.labkey.org/home/Documentation/wiki-page.view?name=externalSchemas

An option worth investigating in your case: PostgreSQL supports "foreign data wrappers" for many database servers, including MySQL, https://wiki.postgresql.org/wiki/Foreign_data_wrappers I haven't ever used these, but you might be able to use the MySQL FDW to make your MySQL tables act as if there were native PostgreSQL tables. You'd then execute your joins as PostgreSQL queries.

Adam