Last week I gave a webinar on the Seven Key Factors for Agile Testing Success. I answered many of the questions on line during the webinar but there were quite a few that I didn’t get to, so I answered them by email. I thought I’d share them as they are questions that have been asked again and again.
========
1. How do you address the whole team approach when there are multiple (>5) teams contributing to a product?

MyAnswer: There a couple of things that can help here. I think there should be an overall Release Manager who owns the whole product release and can make decisions about the ‘whole’. In Scrum, you should be having Scrum of Scrums (or a daily stand-up) with all of the ScrumMasters, or a representative from each team who can talk about the team’s activities. I also believe that the teams should all be working on the same code base and have seen that work very positively. You get a constant integration happening. During demos at the end of every iteration, invite all the teams so that everyone can see what is being developed so there are no overlaps, or gaps in the product development. There is a new book out “Scaling Lean & Agile Development: Thinking and Organizational Tools for Large-Scale Scrum” by Craig Larmen and Bas Vodde (I haven’t read it yet, but it is on my ‘to read’ list) that might help.
=========
2. Have you applied the whole team approach to an operating system (including its additional levels of testing such as system and solution testing) or just at a unit or application level? This refers both to a 3rd party operating system (like Windows and Linux) where we provide a product that includes the 3rd party operating system. It also refers to a proprietary operating system we are responsible the entire hardware and software stack.

My Answer: The system or solution verification testing tends to be more of a QA responsibility, but I like to get input from the whole team when planning it. Developers often have useful tips for testing early – before the final testing needs to happen which can greatly reduce the time spent at the end. I also really encourage early system testing (if possible) and developers need to be aware of this. The product owners, or customers often have great ideas about testing the solution as a whole, so yes, include them. There are definitely some unique issues that many organizations don’t have, since you are working on the hardware and software. I have worked with embedded systems where we had the same issue (on a smaller scale), and I found it even more important to include all the stakeholders so we all knew what to expect.
==========
3. How useful is monitoring branch-flow code coverage for automated tests?

My Answer: I like to monitor code coverage in some fashion and branch flow is one way. However, I don’t usually go in to depth as the power of the coverage is in pointing out gaps. Code coverage tools do not tell you how good the tests are.
============
4. How are the cases handled where automation has limitations?

My Answer: I’m not sure what limitations you mean, but there are areas that sometimes cannot be automated because it doesn’t make sense. I think there are tools out there for most code bases, and sometimes we just need to explore those options. Manual testing sometimes makes sense if the automation takes a lot longer than running the test manually. There is a trade off though – if the test is core to the system, you may want to spend the time to automate even it is hard to do.
===========
5. How are we going to automate the unit test when the modules are under development? Most of the developers are reluctant to write stub code. What is your suggestion?

My Answer: It sounds like the tests you are talking about are more low level integration tests than unit tests. I recommend reading Gerard Meszaro’s book xUnit Test Patterns (it’s big), but pretty good. Unit tests should be small – verifying the behaviour of some small part of the system the developer is working on. It might be as small as a single object or method. Agile cannot succeed with out good unit tests, and it sounds like the developers may not understand their part in the building the quality into the system.
===========
6. If the iteration is for 6 weeks with development activities for 4 weeks, what is the best approach to test the iteration for short time i.e. for 2 weeks?

My Answer: I believe 6 weeks is too long for an iteration. Testing and coding should go hand in hand all the way through. In a two week iteration, I like my first story out of development by day 4 at the latest (preferably earlier) so I can start testing it. It sounds like you are doing a mini-waterfall… code, then test. The faster the feedback to the developer on a story, the easier it is to fix. I encourage small stories (and this might be hard at first), that can be tested right away.
===========
7. What is your definition of defect density? Would you use all test phases in the test pyramid to measure defect density?

My Answer: I actually haven’t found it useful to track defect density on an agile project. If everyone is trying to produce quality code and we are fixing bugs as soon as they are found, the tracking becomes moot. We want to focus on prevention, not detection. However, if you have a legacy system I would track defect density by component so we could determine where a new story might be needed to address the technical debt.
===========
8. How do you handle inter-dependant modules? We cannot release the modules to the customer though they are on high priority, if they are dependable on some other module.

My Answer: High level release planning should take care of dependencies like that. Understanding the risks associated, which includes dependencies is critical. I answered more in Question 1
===========
9. Will agile work well in a team environment where team members work on multiple concurrent projects?

My Answer: Simple answer – No. I think that task switching is very detrimental to the success of an agile project. Now, sometimes a team member has 2 jobs. For example, testers often need to be caregivers for the test environments which is not a full time job by itself. When those testers are on an agile team, their availability is just lower. They need to be aware of their other responsibilities when taking a testing task and may not want to be on a critical path. If a tester is on two different projects or in a support role, their priorities are subject to change at any time and severely impact the amount of time they can dedicate to a project – which in turn means some stories don’t get tested, or tested poorly.
==========