Application Testing For IT Projects
Background
Recently (from mid 2002), I have been involved with some very large IT projects that involve large scale bespoke development done by 3rd parties. Part of my role in these projects has been to provide assurance on the testing processes and managing the direction of the external suppliers.
The contents of this page are the summary of some of my thoughts and learning in this area in an attempt to capture it for future projects.
Things to do
This is a list of some of the things to think about and do when managing the testing on a project.
- Identify and appoint test managers for each phase of testing and one for testing overall (which may be at program rather than project level).
Both customer and supplier should do this but the overall manager should be appointed by (and report to) the customer (even if it is an external contractor) - Create a timeline summary of when inputs, outputs and external milestones are expected
chase them up appropriately (this will be your personal mini project plan) - Identify and communicate with key stakeholders
for each test phase - Have a regular (probably weekly) meeting between testing managers (and assurance managers if you have them)
to report on progress. This means having an agreed measure of progress for every phase of testing - Have regular (probably daily) meetings to discuss defect issues (severity and priority setting, clarification of issues).
Customer and supplier test representatives and development representatives should be involved with this - Test environments should be managed in a transparent fashion
That enables all stakeholders to see when environments are in use, each phase of use should have an owner responsible for configuration, the customer should arbitrate the use of environments and not the supplier
Environments for later test phase should be audited when set up to ensure that they match expected settings.
Wherever possible, environments should be reused so organisations should aim to keep all new systems on environments as similar as possible to allow for the reuse of test environments - Any tools and environments required for testing should be clearly defined in the strategy or entry criteria for each phase
Well ahead of when they are required and procured accordingly. Multi-supplier environments will need to seek a joint agreement on the use of appropriate tools (e.g. should be the same tools for all suppliers, especially for defect management, measurement and reporting). - People from later phases of testing should be involved with earlier phases
so as to obtain early views of the system and architecture and should be able to help shape the testing to smooth the transition to later phases. Similarly, people who will be involved with operations and support should have some involvement from early on to familiarise themselves with the new system - Don't forget to involve external groups early on in design and planning
as few systems today will stand on their own and are likely to need to integrate with other systems (which will have their own development lifecycles). - All testing should take a risk based approach
So that the customer is able to understand, in business terms, what the risks are to not undertaking or shortening a phase of testing - Get multiple vendors in the same room
From the outset, when working with multiple vendors, you need to get them face to face. Get them talking and sharing, better still, get them to share and swap some resources. - Make sure that core test data is shared across all test phases
Especially important for large, multi organisation projects. Test data should be driven from the "least manouverable" environment (e.g. legacy). Sharing test data makes it more likely that test scripts can also be shared and that different phases of testing can actually compare results.
Testing Phases
Here is my take on the various phases of testing needed for a large development project
Feb-2004 - This section needs an update.
- Unit & Link
Who: Usually done within the development teams, customer involvement will be limited but it is recommended that the customer performs an independent check on the quality of the code and internal processes. Where: Development environments What: Check that modules can be called without error. - System
Who: System testing will usually be performed by the supplier doing the development as close ties and scheduling are required.
During this phase, it would be wise (read necessary) to include people who will be involved in later phases. They would help to shape the system tests so that early assurance could be gained. Implementers should also be involved at this stage to gain early experience of operationsWhere: System Testing Environments. Several instances will often be required. What: The customer must be able to measure progress through this phase so good reporting and regular meetings are essential. See the section below on outputs for detail on reporting and tracking. This may well be the longest phase of testing.
System test environments should have access (at least at some point during testing) to as many external systems as practical in order to hunt out teething problems as early as possible.- User Acceptance/User Functional
Who: Lead by customer Where: Model Office What: This must be conducted by the customer and should take place in a "model office" which is reasonably representative of the live environment. It should also use at least some people, preferably from the user community, who have not yet had any exposure to the system. This phase should be very short as it should be a regression type test of key functions, indeed, it may not be required if system testing has been able to provide sufficient assurance. - Business Testing
Who: Lead by the customer Where: Model Office What: Ensure that the new/updated system supports the actual operational business processes and assumptions. Ideally, this takes place in a "model office" and with at least some front line business staff who have not experienced the system before. Training materials should also be tested here. - Co-Existence
Who: Operations (the people who will be supporting and running the live system) Where: Live like, model production. What: Where shared environments will be used, it may be necessary to do joint simultaneous testing to ensure that all systems co-exist happily. - Performance
Who: May be lead by different areas at different times. usually be performed by one or more of the suppliers. Where: Live like, model production. Though early tests may take place elsewhere. A disaster recovery environment may possibly be used. What: This phase may overlap many of the others and will probably happen in parallel (assuming suitable hardware environments are available.
Note that this does not require (necessarily) full scale hardware. Modelling may be used to look at assuring some aspects. The models should then be carried forwards to live support testing where they should also be updated with live running information.- Implementation
Who: Operations Where: Live like, model production and live environements What: This has two aspects. The first is to ensure that the documentation, processes and tools used to implement the system and its environments work corectly. The second is to ensure that the live network (connectivity), API layer and application layer, etc. work prior to go live. To ensure implementation goes smoothly, the live environment should be subject to testing as early as possible. This testing would include network, API layer and application layer connectivity tests to prove that all of the components are talking to each other. Implementation testing can also be extended right back to early phases of the project by applying testing to each environment as it is built. This should aim to get more formal for each phases test environment with lessons learned from previous phases being applied. - Field Acceptance Test
Who: Lead by the customer, all parties involved Where: Live What: A short set of tests done immediately before the users are let loose on the live system (typically at the weekend). The tests will have a few test cases that may be mapped onto live input or will be regressed after completion of the tests (dependent on the system, occasionally this may not be possible and live cases will need to be held back and put through the system). - Live Support
Who: Where: What: Testing will also be required for fixes and support problems after implementation and ongoing test environment is required for this. Fixes in live support are typically fast turnaround and so require rapid, limited testing. - Business Continuity
Who: Where: What: This should test both the physical infrastructure and the operational procedures for handling a major disaster. Typically it is done once the new system has had a chance to stabilise.
These phases have been presented in roughly the correct timeline though pressure of time may well cause them to overlap. It is, however, better not to overlap them if possible so that clear project decisions can be taken based on the outputs from the previous phase.
Outputs for each phase of testing
Each phase needs to produce certain outputs and has certain inputs. These should be summarised in the contract in the form of entry and exit criteria.
- Strategy
Should describe why this phase of testing is needed, primarily in business terms (not technical) - Critical Success Factors
What specific business requirements need to be tested in this phase. Depending on the phase, some CSF's may be non-functional (e.g. performance factors, 1000 users can log on within 10 minutes, etc.) or functional. The functional CSF's are likely to stem from the high level business requirements and the Use Cases1, the non-functional CSF's are likely to come from agreed service level agreements (SLA's) and/or non-functional requirements documents. Note that the testing may prove or disprove the CSF and can still be considered successful - Success Criteria
This is the first output that is primarily technical in nature, the success criteria detail what tests will be performed to prove/disprove each CSF. It may be necessary for the customer to perform their own fitness check of the tests against the CSF's - Test Plan
This essential document details when each test (or at least each criteria) is due to run. It should be baselined as quickly as possible so that changes to it are controlled and impacts understood by both the customer and the supplier. Key elements of the plan may form part of the milestones in the contract with the supplier. The plan document should be more than just a Microsoft Project table, it should offer instructions and guidelines to testing staff on how to implement the test strategy. The project plan itself should follow the principals of goal and deliverable lead project planning (most commonly embodied in the Prince II project management methodologies). - TESTING
- Progress Reports and meetings
Hopefully obvious! These are probably required weekly to begin with but daily in the closing stages of the phase. Do not forget that progress reports need to go to all stakeholders. Progress through a phase must be measurable and the measures should be agreed at contract level - Statistics
Especially important for long phases of testing (several weeks or more). Test managers need to agree what is a sensible set of METRICS (measures) in advance. The customer should not rely on the supplier to create the statistics but should get the underlying measures for themselves. Also, statistics are of no use unless someone is monitoring them and taking actions if anything seems to be going astray. The statistics will be of most use during the middle part of the phase, at the start and end they will be less accurate - Test Results
This output should include a management summary, a summary of findings against the CSF's and sufficient detailed results to allow the customer to check the findings for themselves (the key word here is EVIDENCE) - Review
A review should be conducted at the end of each phase of testing. It should have the following outputs:- List of fixes and implementation improvements
To be applied to environments used in later phases - Lessons Learned
These need to be communicated to later test phases and to other similar projects and programs - Confirmation that all Critical Success Factors were met
Or reasons why not. This will need to be communicated to project managers and possibly commercial managers for further discussion and analysis.
- List of fixes and implementation improvements
Each of these outputs is required and should form key deliverables of any contract between the customer and the supplier.
However, where suppliers wish to use their own (or 3rd party) development methodologies, it may be sufficient to have a document that maps these outputs against the equivalent deliverables specified by the methodology.
Contract Issues
Customers need to be clear about their expectations in their contracts with suppliers. Key to getting value in the testing areas is ensuring that the supplier must provide evidence and measurements at each stage of testing.
You should also use the contracts to enforce different suppliers to work together and share resources. This applies especially to getting the most appropriate people to work on each test phase, getting shared data and shared test scripts. There should also be a common set of tools and they need to be specified in all contracts.
How Big is my System?
It's easy to fall into a trap of thinking of the size of your system in terms of user numbers but in reality the key measure is generally the number of business transactions being processed in a set time period.
This should be broken down into technical transactions whos performance can be consistently measured.
To automate testing, user volumes will need to be estimated as will user "think times", the time an average user will spend thinking between server transactions.
How Far Should I Test?
It is generally accepted that, as far as performance is concerned, less than about 30% of live business volumes will not give realistic results. This will, however, largely depend on the complexity of your system and its data. For example, transaction based mainframe systems may need little in the way of performance testing; rather relying on modelling since the performance characteristics may be relatively easy to predict. On the other hand, complex, multi-environment distributed systems may be almost impossible to model reliably and would require much more actual testing and a larger test environment. Clearly this has a significant effect on costs and may change the cost benefits of mainframe vs. distributed systems.
- Use Cases were defined by Ivar Jacobson in his book "Object-Oriented Software Engineering" and are widely used in large scale application development. Try looking on www.usecases.org for additional information (usecases.org seems to have disappeared! Fortunately the content can be found on http://alistair.cockburn.us/usecases/usecases.html)
