Year 2000 Testing Methodology WG2-0005 Elmar Roberg - Business Transformation Services CSSA/PMI Y2000 SIG WG2 http://www.cinderella.co.za/y2000tes.txt Last update 1998-02-15 1 Introduction Year 2000 testing is one of the most critical activities that any Year 2000 project will engage in. Practice has shown that most companies are rather poor at testing. The average system will be delivered with only around 95% of "bugs" removed. In the case of Year 2000 projects, this means that these bugs will only be discovered after the 31 December 1999, with the consequent disruptions to business and associated problems. When modifying code, most organisation will inject one new defect for every 14 they fix (the best organisations achieve 1:50 and the worst 1:5). Many research organisations indicate that many enterprises will spend 60% of their total Y2000 effort on testing. It is therefore critical that proper attention be paid to the planning and management of testing. This is especially true because testing is the last major activity to occur and time will in all likelihood have run out - i.e. there will be no time for recovery from lost opportunities or unavoidable error. 1.1 Terminology CCI (Compliance Configuration Item) Portfolio Platform System Application COTS 1.2 References IEEE 2000.2 Recommended Practice for IT Year 2000 Test Methods. ISO 8601:1988 Data elements and interchange formats - information interchange - representation of dates and times. Beizer, Boris. Testing and the Year 2000 Problem. American Programmer, Vol10 No8, Aug97. Cutter Information Corp. ANSI/IEEE 1008-1987 Software Unit testing. ANSI/IEEE 1012-1986 Software Verification and Validation Plans. 2 Y2000 Testing Process We will describe the testing-related activities that should be executed during each of the phases of a year 2000 project in this section. 2.1 Awareness There are no testing related activities envisaged during the Awareness Phase. 2.2 Assessment Since management is requested to approve a strategy for Y2000 compliance remediation at the end of the Assessment phase, it is important that the risks are clearly defined for review. Associated with risk, testing must be performed to "prove" that the enterprise will not be subjected to excessive business risk (beyond what it has normally accepted and lived with in the past). Since there are broader issues involved than just the correct operation of systems (for example, there is quite possibly an issue of "due diligence" that affects directors and officers of the enterprise, and there is the likelihood that auditors will be require to report on the effectiveness of Y2000 remediation exercises), it is important that management be made aware, and have the power to influence, the plans to prove the remediation work. It is thus important that a Test Proposal and High Level Test Plan be completed before the conclusion of the Assessment Phase, and that these documents be included as a part of the Assessment Phase exit criteria.2 2.2.1 Create Test Proposal 2.2.2 Describe Hi-level Test Strategy 2.3 Remediation During remediation, whilst the changes are being made to the system, is the time when detailed test cases should be developed. Ideally, a separate team should be working on this. They can act as check on the remediation team to ensure that the most important objectives are being achieved. If the remediation team develop the test cases, there is a chance that the time pressure will seduce them into cutting testing, or developing test cases that prove what they have done (rather than proving what they should have done). The remediation team should be responsible for unit testing of changes, limited integration testing (for example, at the function level) and regression testing (proving that the changes have not changed existing functionality, or introduced new errors. Test beds should be readied during this phase. 2.4 Validation The validation phase should be restricted to Y2000 testing. The test beds should be ready, configuration management in place to manage the relationship of systems in production and test, etc. The testing of each system should be concluded with an acceptance test, where users take back control of the fixed systems. 2.5 Implementation Implementation should not include any testing activities. 3 Y2000 Testing Project Deliverables 3.1 Testing Proposal Base assumption: it will not be possible / may not be necessary to test everything. Test Proposal is created during Assessment Phase to propose to business what, why, how, where, when testing should be done, and by whom. Dec'97 Cap-Gemini survey indicated that all surveyed companies would perform partial testing only. Purpose: to show the decision-making process involved in determining what should be tested, and what not. Reason: for the sake of "due diligence". Test Strategy applies to those systems that will be considered for testing. (Note this important distinction!) Test Proposal should describe how different systems may be grouped for portfolio testing (note that a system may be a part of more than one portfolio) as well as the objectives of enterprise testing. Proposal assumes a risk driven approach. Generic strategies are: - Prevention: effect Y2000 changes. Perform extensive Y2000 testing to validate changes and verify that all Y2000 problems have been discovered. - Retention: accept the risk and deal with it when it occurs. This approach would be used if the cost of avoidance were greater than the cost of risk occurrence. - Detection: by introducing control mechanisms that will give early warning of the occurrence, thus allowing contingency action. Define what contingency action should be taken and test that the contingency action will minimise / remove the risk. - Transfer: by insuring against the risk, passing it outside, for instance having the work done by contractors. - Reduction of effect: fix the most important faults, creating fallback or alternative procedures. Test that reduction of effect will be achieved. - Recovery: define fallback procedures. Test the effectiveness of the fallback procedures. The Testing Proposal should include the following components: - Inventory of systems to be tested - Inventory of systems excluded - Rationale for exclusion of systems from scope - High level Testing Strategy, which would include: - General approach to be used - Test Bed requirements 3.2 Testing Strategy For those systems that will be subjected to testing, describe what, why, how, where, when this testing will be done. I.e. the test strategy would be a refinement of the Test Proposal, for the subset of systems that will be included for testing. A Test Strategy is typically determined for each system, or portfolio of systems to be tested. Partition on the basis of what can conveniently be handled by one team. Note that the remediation team would typically be involved in the testing effort. This should make scheduling interesting. Testing should generally be restricted to date-related testing. There is unlikely to be enough time and resources to do more. The test strategy should address the following aspects: - Assessment of the type of system and system elements. - Description of the major dependencies. For example, "hub" software should be tested before systems around the hub are tested. Platforms should typically be certified before testing of systems that run on them are tested, to ensure that no unnecessary complications are introduced. Systems that have time dependencies should be noted. ú Determination of the factors critical to the validation that testing successfully completed. ú Determination of the testing scope, timing and resources (people, test bed, etc). ú Evaluation of test effort tradeoffs (risk, schedule, scope, quality, techniques, resources, cost). Test Strategy is typically made up of the following components: - Test Plan - List of Functions - Refined Test Bed Requirements - Acceptance Criteria 3.2.1 Test Plan The Test Plan describes the scope, approach, resources and schedule of intended testing activities, as well as the testing environment. The Test Plan describes the specific functions, transactions and features to be tested, who will do the testing, how they will do this and when, and what contingency action will be taken should major risks be realised. The nature and comprehensiveness of testing should be tailored to the criticality of the item. (See the Classification Codes for more information on determining criticality, impact and approach.) It is often useful to make use of a network-diagramming tool (such as may be found in a project management tool like MS Project) to describe the dependencies and flow of a major testing activity. 3.2.2 Functions During the Assessment Phase, an Inventory of relevant (date- impacted) Business and Technical Events should have been created. Events result in processes that are supported by Business and Technical Functions. Inventories of these, together with Function Summaries for relevant functions should also have been created. Business and Technical Functions should have been categorised according to the degree of date impact and the criticality to the enterprise or business unit. Critical Functions should be reviewed to determine the nature and level of testing that should be performed. Test cases would be devised for these functions. Functions are performed by executing transactions. Thus, test cases would describe the event that causes a transaction to be executed, followed by a detailed transaction flow until the condition arises that causes activity to stop. For each relevant function, identify whether (or not) the function will be included for testing and the rationale behind the decision. 3.2.3 Acceptance Criteria There are generally two types of criteria: generic (which apply across the board, and should be considered for test cases in all test plans) and specific (which apply to a specific application). Examples of generic criteria include testing for century change, leap year, etc. Examples of specific criteria include for specific action that would relate to specific application-determined dates, century- spanning, etc. 3.2.4 Refined Test Bed Requirements As the nature and comprehensiveness of the testing is clarified, it is possible to be more specific about the exact requirements of the test bed. Information required for the provision of the test bed includes: Hardware (type, capacity, configuration, throughput, level of availability required, location) Instrumentation tools (if available, required facilitate the validation of tests) Simulators (if available, used to simulate a production environment, setting dates forward, ageing data, etc.) Other software (configuration management tools that will facilitate the management of data sets and test cases, etc.) Data sources (what, how much, where from, etc.) 3.3 Test Documentation Test Specifications describe in detail how testing will be executed. Variations may occur depending on the level of certification that is required. This would typically depend on the criticality of the function and the level of Y2000 (or date) impact. 3.3.1 Test Case A Test Case specifies the set of test data, together with the associated procedures developed with a particular objective in mind. Test data includes a populated database, input values and expected results. The amount of data will depend on the nature of the testing (whether an aspect of performance is included, possible condition coverage, etc). Test cases must be defined for both internal interactions of a system, as well as interaction with other systems (interfaces). In some instances, Pass/Fail criteria may be defined. For example, if a range is acceptable, the low and high values need to be specified. The objective of the test case should be clearly defined (why this test case should be executed). Test conditions should be described, which define different scenarios that would apply (for example, century rollover, century spanning). In addition, any constraints should also be defined (time constraints, dependencies, etc). The test procedures should list the exact steps that need to be followed to achieve the desired level of certification. Expected test results should be identified that can be used to verify a successful test. 3.3.2 Test Log A test log should be used during the course of applying test cases to ensure that every test step is executed according to plan. Small incidents that are resolved immediately by the testing team, and do not require additional follow-up should be recorded on the test log. 3.3.3 Test Incident Report A test incident report should be completed for every incident that occurs that requires additional action. Small issues that are resolved immediately can be included in the test log. 4 Reducing Testing Time Considering the comment made in the introduction about the risks associated with testing (that there will in all likelihood be no time for error), it would be useful to consider some ways in which the elapsed time could be reduced. Remember that there are usually benefits associated with risk taking (else why would you take the risk?). The key is to ensure that one receives the benefit, and is not punished (by realising the risk). - Attempt to do as much as possible in parallel. Consider sequential processes in the testing plan. Is the sequence absolutely required? Can one, with a little extra effort, create separate test beds to test in parallel (i.e. create input data without waiting for the predecessor to produce the required output). Note however, that this will increase cost, also that the amount of checking (of output) will have to be increased and be more careful. - Spend time during remediation to set up the test bed, especially the data sets. - Upgrade hardware to reduce test case turnaround time. - Is it possible to reduce the test case process time? For example, by accepting risks for the benefit of short testing. One would obviously need to consider the criticality of the function. - Introduce formal inspection and review processes into the remediation process. Many of the best organisations have been able to find and remove more than 95% of defects through design and code inspections. The less defects there are, the quicker testing will be. - Reduce project size. The highest productivity is typically achieved by teams of less than ten people in projects of three to six months. Try to cut the suit to fit this cloth. - Test case design and test database design can start as soon as the following has been established: function Y2000 impact and criticality. Elmar Roberg - Business Transformation Services CSSA/PMI Y2000 SIG -WG2 (last updated 1998-02-15)