I have made some progress.

The file "service.bat" in the ftpserver\bin folder has a couple of lines similar to:

set PR_JVM=%JAVA_HOME%\jre\bin\client\jvm.dll

However, there is no "jre" folder in this path in my LabKey installation.

When I change this to:

set PR_JVM=%JAVA_HOME%\bin\client\jvm.dll

And then remove and re-install the service using the service.bat file, then the service is able to start.

In my installation, the "jre" folder is "...\LabKey Server\jre1.5.0_11", and the JRE_HOME and JAVA_HOME variables equate to this folder. The next folder below this is "\bin", not "\jre\bin".