MilestoneSix Quality Assurance Report

Evolving Status Report (See JUNitTestPlan)

Notes:
  • See BugZilla
  • We use enhancements to track work items; they are not bugs, they are reminders of work to do.
  • Some of our low priority items are not to be fixed during the scope of our project, they are reminders for future developers.

Date: May 9th, 2005.

Progress Tests

Test Group Number Test Group Name / Description # of tests written # of tests that can run # of tests that pass
1 Palette tests 14 2 2
2 Properties tests 9 5 5
3 Behaviour tests 18 5 5
4 Contextual Menu tests 13 0 0
5 Stub action tests 6 0 0
6 General non-editor tests 11 0 0
  Total 71 12 12

Command tests

  • As of May 9th, 2005, we have created tests for 15/21 of our commands.
  • All of our command tests pass. We do not know how many of these tests we will have when we are done.

Bug reports

Level Bugs Found Bugs Fixed Bugs Remaining
Emergency 3 3 0
High 11 8 3
Low 15 9 6
Enhancements 30 6 24


Date: May 19th, 2005.

Test Group Number Test Group Name / Description # of tests written # of tests that can run # of tests that pass
1 Palette tests 14 2 2
2 Properties tests 9 5 5
3 Behaviour tests 18 5 5
4 Contextual Menu tests 13 0 0
5 Stub action tests 6 0 0
6 General non-editor tests 11 0 0
  Total 71 12 12

Command tests

  • As of May 19th, 2005, we have created tests for 14/18 of our commands.
  • All of our command tests pass. We do not know how many of these tests we will have when we are done.

Bug reports

Level Bugs Found Bugs Fixed Bugs Remaining
Emergency 4 4 0
High 25 22 3
Low 30 14 16
Enhancements 30 7 23

Requirements

In scope requirements (from MustDoReport)
Status Count
Approved 9
Started 13
Implemented 14
Completed 7
Total 43

Out of scope requirements:
Status Count
Approved 58
Started 1
Implemented 8
Completed 0
Total 67


Date: May 25th, 2005.

(removed one test)

Test Group Number Test Group Name / Description # of tests written # of tests that can run # of tests that pass
1 Palette tests 14 14 9
2 Properties tests 9 9 7
3 Behaviour tests 18 6 6
4 Contextual Menu tests 13 0 0
5 Stub action tests 6 0 0
6 General non-editor tests 10 2 2
  Total 70 31 24

Command tests

  • As of May 25th, 2005, we have created tests for 21/22 of our commands.
  • All of our command tests pass. We do not know how many of these tests we will have when we are done.

Bug reports

Level Bugs Found Bugs Fixed Bugs Remaining
Emergency 4 4 0
High 31 25 6
Low 32 21 11
Enhancements 39 8 31

Requirements

In scope requirements (from MustDoReport)
Status Count
Approved 7
Started 14
Implemented 15
Completed 7
Total 43

Out of scope requirements:
Status Count
Approved 58
Started 1
Implemented 8
Completed 0
Total 67


Date: May 31st, 2005.

Test Group Number Test Group Name / Description # of tests written # of tests that can run # of tests that pass
1 Palette tests 14 14 12
2 Properties tests 9 8 7
3 Behaviour tests 18 7 7
4 Contextual Menu tests 13 0 0
5 Stub action tests 6 0 0
6 General non-editor tests 10 2 2
  Total 70 31 28

Command tests

  • As of May 31st, 2005, we have created tests for 29/35 of our commands.
  • 28 or our 29 tests pass. We do not know how many of these tests we will have when we are done.

Bug reports

Level Bugs Found Bugs Fixed Bugs Remaining
Emergency 4 4 0
High 32 29 3
Low 37 21 7
Enhancements 43 24 19

Requirements

In scope requirements (from MustDoReport)
Status Count
Approved 3
Started 8
Implemented 22
Completed 9
Total 42

Out of scope requirements:
Status Count
Approved 58
Started 1
Implemented 8
Completed 0
Total 67


Date: June 1st, 2005.

Test Group Number Test Group Name / Description # of tests written # of tests that can run # of tests that pass
1 Palette tests 14 14 12
2 Properties tests 13 13 8
3 Behaviour tests 15 14 14
4 Contextual Menu tests 12 11 8
5 Stub action tests 6 0 0
6 General non-editor tests 10 5 5
  Total 70 57 47

Command tests

  • As of June 1st, 2005, we have created tests for 29/35 of our commands.
  • 28 or our 29 tests pass. We do not know how many of these tests we will have when we are done.

Bug reports

Level Bugs Found Bugs Fixed Bugs Remaining
Emergency 4 4 0
High 32 29 3
Low 37 21 7
Enhancements 43 24 19

Requirements

In scope requirements (from MustDoReport)
Status Count
Approved 3
Started 8
Implemented 8
Completed 23
Total 42

Out of scope requirements:
Status Count
Approved 58
Started 1
Implemented 8
Completed 0
Total 67


Date: June 14th, 2005.

Test Group Number Test Group Name / Description # of tests written # of tests that can run # of tests that pass
1 Palette tests 14 14 12
2 Properties tests 13 13 13
3 Behaviour tests 15 14 14
4 Contextual Menu tests 12 11 9
5 Stub action tests 6 0 0
6 General non-editor tests 10 5 5
  Total 70 57 53

Command tests

  • As of June 1st, 2005, we have created tests for 32/46 of our commands.
  • 32 of our 32 tests pass. We do not know how many of these tests we will have when we are done.

Bug reports

Level Bugs Found Bugs Fixed Bugs Remaining
Emergency 5 5 0
High 45 43 2
Low 48 39 9
Enhancements 47 29 18

Requirements

In scope requirements (from MustDoReport)
Status Count
Approved 3
Started 5
Implemented 8
Completed 26
Total 42

Out of scope requirements:
Status Count
Approved 50
Started 3
Implemented 13
Completed 0
Total 66

Evolution not tracked here. Could be inferred, but lack of time.


Date: July 14th, 2005.

Test Group Number Test Group Name / Description # of tests written # of tests that can run # of tests that pass
1 Palette tests 14 14 14
2 Properties tests 13 13 13
3 Behaviour tests 15 15 15
4 Contextual Menu tests 12 12 12
5 Stub action tests 6 1 1
6 General non-editor tests 10 8 8
  Total 70 63 63

Command tests

  • As of July 14th, 2005, 36 command tests have been created. There should be 42 tests at this point.
  • Some commands are used by others internally (package visibility) and don't need to be tested.
  • Some other commands are marked in BugZilla as must refactor/clean and will be tested once refactored/cleaned.
  • All of our programmed command tests pass.

Bug reports

Level Bugs Found Bugs Fixed Bugs Remaining
Emergency 5 5 0
High 47 46 1
Low 55 46 9
Enhancements 68 40 27

  • Of the remaining bugs, only five are in scope. Rest are for future work. They should be completed by the milestone date.

Requirements

In scope requirements (from MustDoReport)
Status Count
Approved 0
Started 1
Implemented 0
Completed 42
Total 43

  • The only remaining requirement is ReqElemStubActions's more esoteric behaviour. Requirement was too general anyways smile

Out of scope requirements:
Status Count
Approved 49
Started 1
Implemented 0
Completed 15
Total 65

QA Postportem

  • Quality assurance was our Achilles heel. troy_achilles_brad_pitt.jpg

  • We rarely implemented any tests before actually doing the coding
  • Our evolving status reports shows snapshots of results right after we implement a bunch of tests.
  • Our testing framework is, in my opinion, very impressive. See CommandLinePDEJunit.
    • However, we underused it.
  • We found lots of bugs using our tests, but we could have found alot more had we given more effort to testing. By leaving our tests for the end, we ended up testing everything manually and then adding a few spot checks to make sure it works. This is not very useful.
  • Our command tests are really useful, our progress tests, alot less. For this reason, we've dropped some of them. The hard part about progress tests is that we sometimes ended up trying to test the framework instead of our code. Hooking up to the appropriate location is the hardest part.
  • Our code coverage must not be very good; don't know it exactly because Clover crashes when compiling our project.
  • I recommend that further developers upgrade the testing framework to make them more useful.
  • We were planning on doing code reviews weekly; we ended up doing only one.
    • Instead of having all team members participate, one member went through all the files to cleanup the code and document.
    • We did not manage to finish on time, the views package has not been cleaned.
    • We found tons of refactoring issues that are apparent once the coding is done, that we could have caught much earlier. They are now enhancement bugs in BugZilla but they could have already been fixed.

  • I'm not one to make excuses, but there are two reasons for our unimpressive results concerning testing
    • Testing an Eclipse plugin is hard. Running the tests in cruisecontrol took us 20+hours. Even now, our framework (took 15+ hours of tweaking for setUp() and tearDown()) seems to have issues with how elements are initialized. We try to access things that are usually available, but fail miserably.
    • Our Quality Assurance Manager did absolutely nothing in the last four months. Beforehand, he performed little or no QA tasks, the other team members had to do them for him. His efforts would have been crucial in ensuring a quality product.

-- JasonKealey - 14 Jul 2005
Topic revision: r1 - 14 Jul 2005, JasonKealey
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback