More Related Content
What's hot (20)
Similar to Software automation (20)
Recently uploaded (20)
Software automation
- 2. What is Test Automation • Developing software to test the software is called test automation. • Test automation can help address several problems. 1. Automation saves time as software can execute test cases faster than human do. 2. Test automation can free the test engineers from mundane tasks and make them focus on more creative tasks. 3. Automated tests can be more reliable. 4. Automation helps in immediate testing 5. Automation can protect organization against attrition of test engineers. 6. Test automation opens up opportunities for better utilization of global resources. 7. Certain types of testing cannot be executed without automation 8. Automation means end-to-end, not test execution alone. • Data generator: Automation should have scripts that produce test dat to maximize coverage of permutations and combinations of inputs and expected output for result comparison.
- 3. Terms Used In Automation • A test case is a set of sequential steps to execute a test operating on a set of predefined inputs to produce the certain expected outputs. • It always have an expected result associated when executed. • It can be documented as a set of simple steps or it could be an assertion statement or set of assertions. • Example for assertion: “Opening a file which is already opened should fail”. • Test case are repeated for several times –build for several times. • Example Log in process is performed in large number of test cases. • In these case the automation code can be reused. • Reusability reduces the effort and time. • The table show two dimensions: – What operation have to be tested – product -specific – How the operations have to be tested – scenarios, framework -specific • When a set of test cases is combined and associated with a set of scenarios, they are called “test suite”.
- 4. Skills Needed For Automation • The skills needed for automation depends on what generation of automation the company is in or desires to be the in the near future. • The automation of testing is broadly classified into three generations: • First generation : Record and Play back: – Avoid the repetitive nature of executing tests. – A test engineer records the sequence of actions by keyboard characters or mouse clicks and those recorded scripts are played back later. – Drawbacks: • Hard coded values so difficult to perform general types of tests. • Requires a lot of manual intervention to detect and correct error conditions. • When the application changes, all the scripts have to be re-recorded, thereby increasing the test maintenance costs. • Second generation: Data-driven: – It helps in developing test scripts that generates the set of input conditions and corresponding expected output. – Test can be repeated for different input and output conditions. • Third generation: Action-Driven: – It enables a layman to create automated tests. – All the actions that appear on the application are automatically tested, based on a generic set of controls defined for automation. – The set of actions are represented as objects and those objects are reused. – Everything relevant to the action are automatically generated. • Input , output, scenarios, framework – It involves two major aspects “ test case automation” and “framework design”.
- 5. What to Automate, Scope of Automation • The specific requirement may vary from product to product, situation to situation and from time to time. • Tips to identify the scope of automation: 1. Identifying the Types of Testing amenable to Automation: 1. Stress, reliability and performance testing 2. Regression test 3. Functional tests Tumb rule : effort for automation does not exceed the effort needed for executing those test case , then it can be automated. 2. Automating areas less prone to change: – Automate the requirements where there is less or no change. – Normally change in requirements cause scenarios and new features to be impacted, not the basic functionality of the product. – User interface normally undergoes changes. – While automating functions involving user –interfaces and non-user interfaces elements ,clear demarcation and “pluggability” have to be provided. 3. Automate tests that pertain to standards: – One of the test that the product may undergo is compliance to standards. – Automating for standards provides a dual advantage: • Test suites developed for standards can be – Used for product testing – sold as test tools for the market. – Certification testing: the certification suites are executed every time by the supporting organization before the release of hardware and software.
- 6. 4. Management Aspects in Automation: – Prior to starting automation, adequate effort has to be spent to obtain management commitment. – Management permission are only given in phases and part by part. – Return on Investment. – Automate the critical and basic functionalities of the product first.
- 7. Design and Architecture For Automation 1. External Modules: – TCDB and Defect repository. – The test cases, the steps to execute them and the history of their execution are stored in the TCDB. – Defect DB contains the details of all defects that are found in various products that are tested in a particular organization. – For automated test cases, the framework can automatically submit the defects to the Defect DB during execution. 2. Scenario and Configuration File Modules: – Scenarios- “how to execute a particular tests” – Configuration files – set of variables used in automation. – The variable could be for the test framework or for other modules in automation such as tools and metrics or for the test suite or for a set of test cases or for a particular test case. – The values of the variables in the configuration file can be changed dynamically to achieve different execution, input, output and state conditions. 3. Test Cases and Test Framework Modules: – Test case is an object for execution for other modules in the architecture and does not represent any interaction by itself. – Test framework – “what to execute” and “how they are executed”. – The test framework is considered as the core for the automation design. – The test framework can be developed by the organization internally or can be bought from the vendor.
- 8. 4. Tools and Results Module: – Build tools. – When a test framework executes a set of test cases with a set of scenarios for the different values provided by the configuration file, the results for each of the test case along with scenarios and variable values have to be stored for future analysis and action. – The achieve of results help in executing test cases based on previous test results. 5. Report Generator and Reports/Metrics Modules: – Once the results of a test run are available, the next step is to prepare the test reports and metrics. – Preparing reports is a complex and time- consuming effort and hence it should be part of the automation design. – Customized report – very high level status, technical report – moderate level of detail of the test run and detailed or debug reports- which are generated for the developers to debug the failed test cases and the product. – The periodicity of the report is different such as daily, weekly, monthly and milestone reports. – The module that takes the necessary inputs and prepares a formatted report is called a report generator.
- 9. Generic Requirements for Test Tool/Framework • Requirement 1: No hard coding in the test suite: – One of the important requirements for the test suite to keep all variables separately in a file. – Variable are called as configuration variables and the file in which all variable names and their associated values are kept is called configuration file. • Requirement 2: Test case/suite expandability: – In a product scenario involving several releases and several test cycles and defect fixes, the test cases go through large amount of changes and additionally there are situations for the new test cases to be added to the test suite. • Adding a test cases should not affect other test cases. • Adding a test case should not result in retesting the complete test suite. • Adding a new test suite to the framework should not affect existing test suites.
- 10. • Requirement 3: Reuse of code for different types of testing, test cases: – The test suite should only do what a test is expected to do. The test framework needs to take care of “how” and – The test programs need to be modular to encourage reuse of code. • Requirement 4: Automatic set and cleanup: – For each test case there could be some perquisite to be met before they are run. – Each test program should have a “setup” program that will create the necessary setup before executing the test cases. – A setup for one test case may negatively work for another test case. So “undo” the setup soon after the test execution for the test case.-cleanup program • Requirement 5:Independent Test cases: – Each test case should be executed alone. – This requirement enables the test engineer to select and execute any test case at random without worrying about other dependencies. • Requirement 6: Test case dependencies: – There may be a need for test cases to depend on others. – A test tool or framework should help to specify the dynamic dependencies between test cases.
- 11. • Requirement 7: Insulating test cases during execution: – To avoid test cases failing due to some unforeseen events, the framework should provide an option for users to block some of the events. • Requirement 8: Coding standards and Directory Structures: – Coding standards and proper directory structures for a test suite help the new engineers in understanding the test suite fast and help in maintaining the test suites. • Requirement 9: Selective execution of test cases: – A framework may have multiple test suites; a test suite may have multiple test programs and a test program may have multiple test cases. – The framework should enable the test engineer to select a particular test case or set of test cases and execute them. • Example : test-program-name 2,4,1,7-10
- 12. • Requirement 10: Random execution of test cases: – Test engineer may some times select a test case randomly from a list of test cases. – Given a set of test cases and expecting a test tool to select the test case is called random execution of test cases. – Example: random random test-program-name 2,1,5 test-program1(2,1,5) test-program2 test-program3 • Requirement 11: Parallel execution of test cases: – Parallel execution simulates the behavior of several machines running the same test and hence is very useful for performance and load testing. – Example: instances,5 instances,5 test-program1(3) test-program1(2,1,5) test-program2 test-program3
- 13. • Requirement 12: Looping the test cases: – There are two types of loops that are available. • One is the iteration loop which gives the number of iterations of a particular test case to be executed. • Other is the timed loop, which keeps executing the test cases in a loop till the specified time duration is reached. • Example : Repeat_loop,50 Time_loop,5 Hours Test-program1(3) test-program1(2,1,5) test-program2 test-program3 • Requirement 13: Gourping of test scenarios: – The group scenarios allow the selected test cases to be executed in order, random, in a loop all at the same time. – Example : group_scenario1 parallel,2 AND repeat,10 @scene1 Scene1 test –program1(2,1,5) test-program2 test-program3
- 14. • Requirement 14: Test case execution based on previous results: – Rerun all the test cases which were executed previously. – Resume the test cases from where they were stopped the previous time. – Rerun only failed/not run test cases. – Execute all test cases that were executed on “date”. • Requirement 15: Remote Execution of Test Cases: – The central machine that allocates tests to multiple machines and co- ordinates the execution and result is called test console or test monitor. – It should be possible to execute/stop the test suite on any machine/set of machines from the test console. – The test results and logs can be collected from the test console: – The progress of testing can be found from the test console. • Requirement 16 : Automatic archival of test data: – Archival of test data must include • What configuration variable are used; • What scenario was used; and • What programs were executed and from what path.
- 15. • Requirement 17 : Reporting Scheme: – Audit logs are very important to analyze the behavior of a test suite and a product. – A reporting scheme should include, • When the framework , scenario, test suite, test program, and each test case were started/completed; • Result of each test case; • Log messages; • Category of events and log of events and • Audit reports • Requirement 18 : Independent of language: – The framework should be independent of programming languages and scripts. – A framework should provide choice of programming languages, scripts and their combinations. – A framework or test suite should not force a language/script. – A framework or test suite should work with different test programs written using different languages and scripts. – A framework should have exported interfaces to support all popular, standard languages and scripts. – The internal scripts and options used by the framework should allow the developers of a test suite to migrate to better framework.
- 16. Process Model for Automation • Delivery of the automated test should be done before the execution phase so that the deliverables from automation effort can be utilized for the current release of the product. • SDLC model followed for the product development is followed for the automation. • “Test the test first” – an automated test suite has to be tested first before it can be used to test the product. • If test suites report wrong defects, it will severely impact the compatibility of test automation and will hamper further automation efforts. • A set of four activities can be performed at each phase of product and automation development. • The test suites form one of the deliverables from automation. • W model includes two parallel set of activities for development and two parallel set of activities for testing. • The model is only to ensure the flow of activities, there is no binding on start and end date because – The product development and automation are two independent activities. – In many organizations the same test team is involved in testing the product and in developing the test suite.