The more recent builds of selenium will work with Firefox 3, but they haven't made an official build yet. I've been running some old version for a while now without any problems except for the Study BVT running slowly.

You can find the nightly builds of selenium here:

http://nexus.openqa.org/content/repositories/snapshots/org/openqa/selenium/selenium-remote-control/1.0-SNAPSHOT/

Dig around in one of the selenium-remote-control-1.0-XXXXX-dist.zip files to find selenium-server.jar and selenium-java-client-driver.jar and replace your local versions in <labkey-root>/server/test/lib. Before replacing the jars, run "ant stopSeleniumServer" from the server/test directory.

Kevin