Hi Paul,
Unfortunately, we don't have documentation on configuring and using RServe with LabKey yet. There isn't too much configuration required:
1) From the admin console, click on the "experimental features" link and enable the RServe experimental feature.
2) From the admin console, click on the "views and scripting" link and add/edit the R engine configuration to include the hostname, port, remote report temp directory, remote pipeline directory, and the remote user and password if needed. I encourage using an RServe user and password as described here:
http://www.rforge.net/Rserve/doc.html#conf
Once configured, R reports executed by LabKey will be executed on the remote RServe machine by first writing a .r script file into the report temp directory then telling RServe to eval the script. R should probably have the Rlabkey package installed to make http requests back to the LabKey server, but I don't think it is a requirement.
Unfortunately at this time, LabKey server can't be configured to use both a local R and a remote RServe server. All R scripts will be executed by RServe if it is enabled.