The need for testing strategy document

In a recent meeting one of my colleagues asked me why we don’t have a test strategy document. I asked why we need one but I didn’t get a straight forward answer.

I personally do not like to have these kind of documents. I am sure these documents may help in other teams / companies but for the last 14 years these documents were more of a burden than a help or guidance.

First of all, what is a Test Strategy? It is basically defining how we are going to test the application. What sections does it contain? In high level,

  • Document review and approval process or people
  • Scope
  • Test approach or different levels of testing and by whom
  • Environments the application will be tested
  • Tools be used
  • Release process
  • Risks
  • Other process like defect management, etc

Almost all the test strategy documents that I have seen in my previous experiences and the templates online shows a section for definitions of each test type and who own each section. For example, Unit tests by Developers, Feature tests by Developers, Acceptance test by Testers, Performance test by Developers etc along with what is a unit test, what is a feature test etc.

Some can say, this is assigning responsibility. For me it’s encouraging a division in responsibilities. A division between development and testing teams. Becomes a waterfall method and push the responsibilities over the wall. You may disagree with me here. However, my thoughts and my behaviours are a reflection of my own experience.

As modern teams, we expect development and testing functionalities to be working together. Pair when possible. Open discussions and collaboration on how the product quality will be assessed. There for I do not want to introduce these definition documents and encourage the division of responsibilities. Sometimes people find comfort on these definitions and role association. This can be harmful if you are expecting the teams to be working together.

If you are working in a free thinking, dynamic team who know that they should be flexible around definitions and roles and responsibilities, then you can have a test strategy as a guide document. But if your teams are not there yet, there can be confusions of who do what and when and these type of documents can limit their thinking. If not having these type of documents will help the team to be independent and flexible, then definitely that’s where I would go for.

Sometimes teams can get hung with the document and as well as quote the document out of context. For example, in your test strategy document you have defined that test environment should not have live data. If there is a situation, that you have to have live data in on lower environment, someone can say, it is defined in the document and we can’t so we can’t. Deal with the issue in a different way. The different way can cost time and money and the data that is coming from live have no sensitive data. But yet because the document say so, it is not possible.

You may think this is an extreme scenario. It is actually not. I have experienced this in several occasions. As agile teams we are fast paced teams. Sometimes we have to be creative in dealing situations and the whole team should come together. The document can isolate test teams and let the rest of the team go “It’s not our responsibility. We have done what we are asked for”. I have being there, many times. What we say is no finger point but the first question anyone will ask is, how come this slipped testing? Not, how come we missed this as a team? May that question should be documented in the Test Strategy?

If you have test strategy documents, how frequently are you keeping them updated? Or have you created them so generic so the document is valid for all purposes? Once we had a 60+ pages power point document that no one in the team read, or used. But it ticked a box. The management wanted a test strategy and there is a test strategy. Nothing beyond that point.

The document sometime can limit the communication and creativity of the team. Instead of have ad-hoc discussion about how we test something, teams can refer the document and make their own assumptions and interpretations. This also can discourage people to try different tools and techniques. I had being questioned trying out a new tool just because it is not mentioned in the test strategy document. When I update the document, it had to go through the long list of approval process which I had gave up trying the tool.

Does no test strategy document mean, the team doesn’t have a strategy? Team doesn’t know what to test, how to test, what tools to be used and what environments to be used?

I think testing strategy restricts teams and their capabilities. It can depend on what type of a project / product you are working for. If you are working in a critical application you may need one to define all the complexities. Others should learn to collaborate. What’s the word? Be agile about it…

So who should have a test strategy document?

  1. Teams that are new to Agile practices
  2. Teams who have less experiences team members who need guidance over best practices and process
  3. Teams that works in different geographical locations

Above mentioned situations do need a strategy to guide the team and make sure the team is evolving / practicing a defined process. However, these documents should be used for short time until the teams understands the process and value. Then it should be open for discussion and changing according to the team / project / product needs.

My final thoughts to summarise above is,

Testing strategy is Testers should collaborate with Business Analysts on requirements gathering, estimations and story writing. Then collaborate with developers on pair automation while executing their exploratory testing skills. Occasionally use a written document or guideline to support the teams who need direction but later encourage them to start discussions and create team level work methods.

So what about the test plan? I have listed my thoughts about detailed test plans in here.

Advertisements

Test plan : Does it need to be a detailed document?

Recently one asked me does the test plan needs to be a lengthy document.

My immediate answer was “No”.

Because of the puzzled look, I continues.

Lengthy documents used mainly in projects that used the waterfall method. In Agile world, the documentation is minimal. Yes, Test planning is a very important and essential activity that everyone should do. However, it should not be a lengthy document where no one reads or no one have the time to maintain it or yet do not have time to create it in the first place. It should capture the essential functionality that effected. We can use mind maps or a simple Excel table.

Then I saw a smile…

I am a big fan of test plans. In my current gig there is one big product, there are feature implementations, and extensions that happen every year to cater new business potentials and to retain customers. We are in continuous delivery mode where releases happen every week. Yes, short turn around and very happy customer base. When the new features prioritised and up for discussion (requirement gathering or design or estimate) there is a high possibility that these features will not implement in next few weeks. Some could be in months because suddenly work priority changed. (This project structure and agile concepts usage may not agree with you but it works… It works for the project at least. So we will skip all the questions around it) .How do I keep my thoughts over the design/requirements and possible break points?

I do the test planning right after the discussions. It will cover various things such as the local the impact be, what areas will be effected from the implementation, what areas I think will effect with the changes, high level workflow  or user journey, rules that being discussed, what customer base should be effected and not etc. This will be a mind map or a simple table.

When the functionality is up for development, then these test plans help me to gather my thoughts. What my thinking was. I was practicing this for many feature implementations and it always helpful. Because these notes are simple and to the point they are useful to everyone who are involved with that piece of work. I even used few of these artifacts to look back at any time if someone come to me and ask “Hey do you know what we have done in x feature by any chance?”

I may not use the conventional test plans. However, the planning I do works very well for my current project and me. This not only help me to collect my thoughts. I try to learn from others and implement the best practices they recommend. If it works, I continue to use it. I may do adjustments to cater to the project need. Otherwise lookout for more.
It is a continuous learning process and I hope you learn something new today or I have given a success story of a method you wanted to try.

Sample:

Feature name
Tool ABC
Region affected UK, US  Initial development will be focused only for US
Browsers Default list IE7 to IE11, Latest Firefox, Latest Chrome, Latest Safari
Changes to Landing Page Template Change Feature flagged until the MVP is implemented / Navigation template ABC
Initially the content will be only enabled for US
Landing page category show as Modules instead of Categories
Sections Landing Page Logged out view About ABC box hidden
Logged in : unentitled  view About ABC box shows. (Text will be managed via content management tool)
Logged in : entitled  view About ABC box shows. (Text will be managed via EMT)
New and updated Logged out view 10 most recent articles of all articles
Logged in : unentitled  view 10 most recent articles of all articles
Logged in  : entitled  view 10 most recent articles of all articles
Content Modules 3 content module placeholders as default for all Tools
Icon New icon should displayed along side with a new colour
Search ABC tool articles should not be searchable via Main search
Local search box should be available
Search results Should return ABC articles on results
HR Learning Center refine filter should be available
Breadcrumb Breadcrumb should display on ABC tool from landing page onward to categories and articles.
Footer ABC tool should not display under Tools section
Tools Page ABC tool should not list under a defined business category
 Menu ABC tool should not list on main menu
EMT User should be able to add categories to ABC via EMT
Adding article source ABC is not MVP
Article page Only subscribers will be able to view the articles
Download feature should not be available.
S3 resources If S3 resources being mapped in to the ABC articles
Article styles The default functionality should be applied
Print article The default functionality should be applied
Subscribe form Not mentioned on initial requirements.
Request a demo form Not applicable as the functionality is for US
Feedback form Not applicable as the functionality is for US
Entitlements ABC will be an individual product
Product name : ABC_product_name
Tags Standard analytics tags should be included
Article Tags No special article tags
Google site map Should not index on Google. Sitemap should have No index, No follow to the tool