Is it possible for your duplicate row logic to be performed on the client using JS, instead of via SQL? For example, if you could load whatever source data you need to the client, you could potentially very quickly compare your incoming data to that, identify the minimal inserts, and then insert those new rows using the API. If so, the avoids needing the temp table. Otherwise, like Jon says ETL is probably a good option.