I did some reading and learned that R was not running in 64-bit mode through labkey. I went into the admin console, and altered the R engine's command, adding:

--arch=x86_64

this will cause R to run in 64-bit mode and seems to fix that memory issue.