Performance

Engineering
Services

Managing performance for software that is outsourced can be a mine field. As the client you might assume that your contractor is using good software performance engineering practices to develop your software. Unfortunately, all too often the delivered product does not meet its SLAs. This might be because the SLAs were vague or because the contract did not explicitly include mention of SPE. In extreme cases, the contractor might have been misleading about their software performance engineering capabilities.

Avoiding Performance Failures

To avoid performance failures with your outsourced software:

• Make sure that the performance requirements are spelled out explicitly in the contract and that there is enough information to unambiguously determine whether they have been met.
• Make sure that your contracts require contractors (e.g., external developers, suppliers, etc.) to use SPE in developing your products to avoid unpleasant surprises when the products are delivered. The contract should specify the SPE deliverables to be produced during development. It is also important to ensure that those SPE deliverables are assessable (i.e., that you can use them to determine that SPE has indeed been used and used properly on your project). You will need someone in-house who is capable of conducting this assessment independent of the contractor.

Best Practice

The best example that we have seen of including SPE requirements in contracts was the Canadian Forces Supply System Upgrade (CFSSU) Project. In their RFP they specified deliverables, for each phase of the development process, that would ensure that SPE steps would be conducted and that contract monitors could assess the deliverables to determine how well the SPE steps were executed. As soon as the RFP hit the street, our phone started ringing — it was contractors who wanted/needed to learn more about SPE. So the RFP definitely achieved its goal of ensuring that SPE steps would be conducted. The final result was that the Canadian Forces received a system that met both functional and performance requirements. The contractors were pleased because they knew exactly what was expected, and could price extra performance risk management steps into the bid response.

The best illustration that we have seen of why you need assessable SPE deliverables and the skill to evaluate them was provided by a company we'll call "Slow Systems." After repeatedly failing to meet performance requirements, Slow Systems held a symposium for clients and presented their new methodology for meeting customer performance requirements. When asked by one customer if they had used this process on any real projects, the presenter responded that they had used it on many real projects. However, as the presentation went on, it became clear to us that they had not. For example, the presenter stated that once performance models were built, it was no longer necessary to do performance testing during the development process. This is an obvious recipe for disaster, as anyone who tried that on a real project would have learned.

SPE Contract Deliverables

SPE contract deliverables fall into four broad categories:

• Plans: These artifacts are targeted primarily at project management. They include the technical plans for how SPE is applied in each development phase, as well as policies, and procedures governing the production and maintenance of SPE artifacts.
• Performance requirements: SPE artifacts include specifications for key performance scenarios, along with quantitative, measurable criteria for evaluating the performance of the system under development. They also include specifications for the execution environment(s) to be evaluated.
• Performance models and results: This category includes the performance models for key scenarios and operating environments, along with the model solutions for comparison to performance requirements.
• Performance validation, verification, and measurement reports (V&V): This category includes documentation and measurement results that demonstrate that the models are truly representative of the software performance, and that the software will meet performance requirements.

Our extensive experience in performance and software development uniquely qualifies us to help you gain control of the performance of your contract development and prevent problems as new applications are developed.

Contact us today to learn how we can help.