Regression Test Assignment

From Foss2Serve
(Difference between revisions)
Jump to: navigation, search
m (added knowledge unit)
 
(2 intermediate revisions by one user not shown)
Line 51: Line 51:
 
  30% how thoroughly did you test the existing code?
 
  30% how thoroughly did you test the existing code?
 
  20% do your tests verify that each function acts as documented as regression tests?
 
  20% do your tests verify that each function acts as documented as regression tests?
 +
 +
{| border="1" class="wikitable"
 +
! Criteria
 +
! Level 1 (poor)
 +
! Level 2 (fair)
 +
! Level 3 (excellent)
 +
|-
 +
| '''Instructions (10%)'''
 +
| Instructions to fork the repo, only commit the test file, and submit the url were not followed
 +
| Some instructions were not followed precisely, such as committing unnecessary files
 +
| Instructions were followed accurately and precisely
 +
 +
|-
 +
| '''Version Control (20%)'''
 +
| Several mistakes shown in demonstrating ability to fork, commit progress, and pushing to remote (forked) repository
 +
| Minor mistakes shown in demonstrating ability to fork, commit progress, and pushing to remote (forked) repository, such as only committing once or forgetting to push local changes to the remote
 +
| Demonstrated ability to fork, commit progress, and pushing to remote (forked) repository
 +
 +
|-
 +
| '''Unit Tests (20%)'''
 +
| Tests not adequately set up nor separated into individual test functions that each check one specific case
 +
| Unit tests generally follow the correct format but do not follow all conventions such as poorly-named functions or a test function really checking multiple cases
 +
| Unit tests are organized in a collection of test functions which each test specific cases and each test has appropriate input, output, and assertions
 +
 +
|-
 +
| '''Test Thoroughness (30%)'''
 +
| Multiple lines of code and/or branches are not exercised by the tests
 +
| The tests cover most of the code according to gcov, but missed possible branches
 +
| The tests cover all possible branches as reported by gcov's "branch coverage"
 +
 +
|-
 +
| '''Regression Test Effectiveness (20%)'''
 +
| Changing the code with new (or fixed) bugs do not change the result of the tests
 +
| Some corrected bugs result in passed tests and injecting new bugs cause new failed tests but the tests do not correctly report all changes
 +
| Corrected bugs result in passed tests and injecting new bugs cause new failed tests with appropriate names that identify the failing feature
 +
 +
|}
  
 
=== Additional Information: ===
 
=== Additional Information: ===
Line 89: Line 126:
 
[[Category: Learning_Activity]]
 
[[Category: Learning_Activity]]
 
[[Category: Quality_and_Testing]]
 
[[Category: Quality_and_Testing]]
 +
[[Category: Good Draft]]

Latest revision as of 17:40, 8 March 2017

Title

Regression Test Assignment

Overview

Participants write unit tests (in GoogleTest) to establish reliable regression tests for a C++ class

Prerequisite Knowledge

Students need to have installed and programmed unit tests with GoogleTest. They should also know how to run a coverage analyzer and interpret its results. Intermediate knowledge of Git version control is required.

Learning Objectives

Students should write unit tests and use a coverage analysis tool to identify test cases they may be missing

Directions:

Students:

Overview

Your job is to test the GoalUndo class thoroughly so that developers can verify that it is working correctly and have regression tests for when the code is changed during maintenance. That means that your job is not only to verify that the current solution code is working, but that any implementation of the existing functions work as described in the documentation.

Instructions

  • Fork the GoalUndo repo into your own GitHub space
  • Clone your fork onto your own computer
  • Set the upstream remote in case the instructor needs to give you changes to the code during the exam
  • Working within a new branch, add unit tests that thoroughly test the functionality as described in the documentation
  • Use your judgment -- based on what you learned about testing thoroughly -- to decide when you have written sufficient tests to be confident that any correct implementation will pass your tests and that any incorrect implementation will fail.
  • Once you are done testing, push your changes into your fork (keeping it in your branch)
  • Submit the url and branch name of the branch on your fork with your tests
  • Make sure you have only pushed changes to the GoalUndoTest.cpp file. No other files should be changed on your GitHub fork. Any commits to your fork after the deadline will be considered late.

Instructors:

Make your own local clone of the GoalUndo repository and change the code so that it will work as documented in some cases but not in others (such as changing conditional operators or types of control structures). To grade each student, fetch the student's fork and then merge with the changes you made to inject intentional errors into the code. Run the student's tests and confirm that tests fail where they should be testing the functionality broken by the errors you introduced.

Deliverables:

A fork of the GoalUndo repository, with added tests

Assessment:

Grading Rubric

10% did you follow the instructions accurately?
20% did you demonstrate recommended practices with git and GitHub?
20% were unit tests written and formatted following recommended practices?
30% how thoroughly did you test the existing code?
20% do your tests verify that each function acts as documented as regression tests?
Criteria Level 1 (poor) Level 2 (fair) Level 3 (excellent)
Instructions (10%) Instructions to fork the repo, only commit the test file, and submit the url were not followed Some instructions were not followed precisely, such as committing unnecessary files Instructions were followed accurately and precisely
Version Control (20%) Several mistakes shown in demonstrating ability to fork, commit progress, and pushing to remote (forked) repository Minor mistakes shown in demonstrating ability to fork, commit progress, and pushing to remote (forked) repository, such as only committing once or forgetting to push local changes to the remote Demonstrated ability to fork, commit progress, and pushing to remote (forked) repository
Unit Tests (20%) Tests not adequately set up nor separated into individual test functions that each check one specific case Unit tests generally follow the correct format but do not follow all conventions such as poorly-named functions or a test function really checking multiple cases Unit tests are organized in a collection of test functions which each test specific cases and each test has appropriate input, output, and assertions
Test Thoroughness (30%) Multiple lines of code and/or branches are not exercised by the tests The tests cover most of the code according to gcov, but missed possible branches The tests cover all possible branches as reported by gcov's "branch coverage"
Regression Test Effectiveness (20%) Changing the code with new (or fixed) bugs do not change the result of the tests Some corrected bugs result in passed tests and injecting new bugs cause new failed tests but the tests do not correctly report all changes Corrected bugs result in passed tests and injecting new bugs cause new failed tests with appropriate names that identify the failing feature

Additional Information:

Knowledge Area/Knowledge Unit

Software Engineering / SE Software Verification Validation

Topic

Unit Testing, Test Coverage

Level of Difficulty

Writing thorough unit tests will take intermediate knowledge of object oriented programming and predicting possible future bugs will take advanced reflective cognition

Estimated Time to Completion

The assignment should take from 1-3 hours of work and is recommended as an equivalent to a relatively small programming assignment.

Materials/Environment

Need to have a working C++ development environment with GoogleTest and gcov installed

Author

Kevin Buffardi

Source

Kevin Buffardi

License

Licensed CC BY-SA 4.0



This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License

CC license.png

Personal tools
Namespaces
Variants
Actions
Events
Learning Resources
HFOSS Projects
Evaluation
Navigation
Toolbox