This document summarizes the test process and guidelines that LabKey uses to ensure reliable, performant releases of LabKey Server. A combination of developer code review, manual testing, regression testing, automated testing, and release verification ensure ongoing quality of the product.
Specification Phase
- A client proposes a new feature or enhancement and provides a set of requirements and scenarios.
- LabKey writes a specification that details changes to be made to the system. The specification often points out areas and scenarios that require special attention during testing.
- Specifications are reviewed by developers, testers, and clients.
- Developers implement the functionality based on the specification.
- If deviations from the specification are needed (e.g., unanticipated complications in the code or implications that weren’t considered in the original specification), these are discussed with other team members and the client, and the specification is revised.
Testing Throughout Development
- If the change modifies existing functionality, the developer ensures relevant existing unit, API, and browser-based automated tests continue to pass. Any test failures are addressed before initial commit.
- If the change adds new functionality, then new unit, API, and/or browser-based automated tests (as appropriate, based on the particular functionality) are written and added to our test suites before the feature is delivered.
- Developers perform ad hoc testing on the areas they change before they commit.
- Developers also run the Developer Regression Test (DRT) locally before every commit. This quick, broad automated test suite ensures no major functionality has been affected.
- A manual tester (a different engineer who is familiar with the area and the proposed changes) performs ad hoc testing on the new functionality. This is typically 3 – 5 hours of extensive testing of the new area.
- Clients obtain test builds by syncing and building anytime they want, downloading a nightly build from LabKey, retrieving a monthly sprint build, etc. Clients test the functionality they've sponsored, reporting issues to the development team.
- Code reviews are performed to have more developers review the work before it is considered complete.
Automated Testing
- TeamCity, our continuous integration server farm, builds the system after every commit and immediately runs a large suite of tests, the Build Verification Test (BVT).
- TeamCity runs much larger suites of tests on a nightly basis. In this way, all automated tests are run on the system every day, on a variety of platforms (operating systems, databases, etc).
- TeamCity runs the full suite again on a weekly basis, using the oldest supported versions of external dependencies (databases, Java, Tomcat, etc), and in production mode. This ensures compatibility with the range of production servers that we support.
- Test failures are reviewed every morning. Test failures are assigned to an appropriate developer or tester for investigation and fixing.
Issue Resolution
- As an open-source project, the public syncs, builds, and tests the system and reports issues via our support boards.
- Production instances of LabKey Server send information about unhandled exception reports to a LabKey-managed exception reporting server. Exception reports are reviewed, assigned, investigated, and fixed. In this way, every unhandled exception from the field is addressed. (Note that administrators can control the amount of exception information their servers send, even suppressing the reports entirely if desired.)
- Developers and testers are required to clear their issues frequently. Open issues are prioritized (1 – 4):
- Pri 1 bugs must be addressed immediately
- Pri 2 bugs must be addressed by the end of the month
- Pri 3 bugs by the end of the four-month release cycle.
- Resolved issues and exception reports must be cleared monthly.
Release Verification
- On a monthly basis, a "maintenance release" build is created, gathering all feature branch changes. This build is then:
- Tested by the team on the official LabKey staging server.
- Deployed to labkey.org for public, production testing.
- Pushed to key clients for testing on their test and staging server.
- The fourth month of this cycle (March, July, and November) is a "production ready" release with additional preparation steps.
- All team members are required to progressively reduce their issue counts to zero.
- Real-world performance data is gathered from customer production servers (only for customers who have agreed to share this information). Issues are opened for problem areas.
- Performance testing is performed and issues are addressed.
- After testing on staging, this release candidate build is deployed to all LabKey-managed production servers, including labkey.org, our support server, and various client servers that we manage.
- The build is pushed to all key clients for extensive testing beta testing.
- Clients provide feedback, which results in issue reports and fixes.
- Once clients verify the stability of the release, clients deploy updated builds to their production servers.
- After all issues are closed (all test suites pass, all client concerns are addressed, etc.) an official, production-ready release is made.
Related Topics