SWTC:What is testing?

Updated with a link to Richard’s post.

Mark Winteringham (@2bittester) and Dan Ashby (@DanAshby04) had come to this wonderful idea of sharing their experience and knowledge with testers who are new and experienced. This is also a good playground for the experienced to gain mentoring skills.

I went in as a student as I think I still need to learn a lot and I was right.

Mark, Dan and Tony Bruce (@tonybruce77) discussed about what is testing. Everyone’s input went on to the board.


If the list is not readable the items as follow,
•    Asserting expectations
•    Ensuring quality
•    Assuring quality
•    Understanding objectives
•    Making sure it works
•    Finding / preventing bugs
•    Discovering software behavior
•    Test scripts
•    Avoiding blame
•    Analyse risks
•    Making sure product meets objectives
•    Reporting
•    Meeting requirements
•    Gatekeeping

We discussed few items from the board such as,
•    Assuring quality
Question was how we can assure quality. It is a vague task depend on the person, environment and various other facts. Quality is a responsibility of the whole team. If someone gives list of things that needs to be attain to assure the quality, still there be issues as things change.

•    Gatekeeping
Can tester be Gatekeepers? If we say No, the product release be stopped or the management decide to go ahead? If so, why they need a gatekeeper? To fingerprint if something goes wrong. All testers can do is to point the risks and let the management to take the decision.

•    Test scripts
Test scripts / Test cases can be manual or automated. Testing done only using the test scripts/cases. No. Why do we need test scripts? Mostly for the repeating tasks that can followed systematically sounds like automation. Test scripts changes with the functionality changes manual or automate. Maintenance is necessary and it is costly. If you need someone to learn the system test scripts not the right way to follow. Encourage them to do exploratory testing.

Then Mark explained how he would explain testing. He explained how testing is always stay between Expectations and Actual product whereas testers we should try to keep the gap minimal.
Unfortunately, I could not take a picture of his diagram. Sorry Mark.
Dan explained testing with the diagram below.


Information is in focus. Information can be about product design, product idea, requirements, processors etc. Information is the centre of all of it.

Testing is uncovering more information via exploration or investigation. Uncovering information be done by asking questions, explore requirements, explore designs, and join in with discussions. The information that are uncover will inform new test ideas to explore.

Information enable many checking activities and regression checks one of them. When new features introduce to ensure the existing is not broken.  Checks done in a script format that can followed either by a person or by an automated process. The output from checks inform the broken tests as well as it can inform what not tested as well. If we have passing checks that can prove that the information are behaving as it required.

Tony explained how information be used for decision making.


Information gathering will lead to experiments, learning and explorations. Results of that can be collated. This collated information helps the software intelligent. With the software intelligent the business intelligence can be supported which lead to decision making.

This can be going in loop. When business require information to make a decision same process happens again from the beginning.

So the original list of “what is testing” updated.

•    Checking
•    Learning
•    Business decision
•    Communication
•    Uncovering
•    Value to a person
•    Communicate about what we have not done!

Practical session taken place afterwards to go through http://www.drawastickman.com.

Richard Bradshaw (@Friendlytester) was kindly agreed to mentor 3 of us.
This site we have not seen before and had no knowledge. Therefore, we spent about 5 minutes exploring the site to understand its functionality. We came to following conclusions.

  • It’s a game
  • Works in mobile / different browsers
  • Have an app in app store
  • It is interactive
  • Navigation
  • Possible bug
  • Etc (Shame I didn’t take a photo of that list!!!!)

Richard was advising us to,

Richard had written a Mentor Experience Report and you can access the post from here.

I am sure I have missed some, as I was not fast enough to note them all.
At the end, all teams came together and discussed the learning. Following are some.

  • Don’t come to a conclusion without knowing enough information
  • Build a practice / discipline as a tester and do not change because of the project.
  • Note taking/ mind mapping
  • Go through social media to gather testing ideas
  • If you are not happy using the product so will your customers
  • Explore test ideas

So that’s a wrap! Until next time…

ST48 – Testing Challenge : Exploratory testing – an example

With Sydney testers I got the chance to get involve on a practical session on Exploratory testing (ET). Widely known and used but still a question to most.

Host: Rich Robinson from Sydney Testers meet-up group

Following is a summary of what’s being discussed. .

What is Exploratory testing? A definition.

“Exploratory software testing is a powerful and fun approach to testing yet widely misunderstood. In some situations, it can be orders of magnitude more productive than scripted testing. All testers practice some form of exploratory testing, unless they simply don’t create tests at all. ” – JBatch

“Exploratory software testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the value of her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.” –  Kaner

Elaborate more…

ET is about learning the product. We can discover information by testing. We can decide what to tests and we can design the tests. We can execute them and the results help discover new information about the product. It might be good, bad, or neither. This information helps us to decide new tests, and so on.

Furthermore, ET is more about discovering new information by testing, and using your existing knowledge. You can start from knowing nothing, or knowing ‘everything’. It’s not only for tester who do not know the product but for the product expertise as well. Testers who are naturally curious, fast learners, insightful, and really want to find possible bugs will adopt ET easily despite the level of knowledge of the product.

How to apply these concepts practically?

  • Focus an area for testing or identify a purpose. Do not opt to test the whole product
  • Add sections such as the purpose, the approach, goal to achieve, what the focus should be to the testing framework
  • Clearly distinct focus, objective, or mission for your testing
    Example : for a login screen, your focus might be, to login as the wrong person.
  • Assign small parts of tests to each member
  • Time box each session
  • Record the session. You always can use a notepad.

This document is referred as a charter document. Sections added to the charter document are referred as containers. These sections can be filled up later. The mind-set is that you are in full discover and info logging mode.

Example for sections

  • Objectives
  • Information sources : Where do you get information about the item user test
  • Data to be used for the tests
  • Procedure to follow or available
  • Test ideas :  The scenarios worth checking out. A backlog of tests that needs to be executed.
  • Findings : Overall important information found. The summary of your report
  • Questions

More on filling the sections of the charter document…

It could be a question, or observation. It could be the data you are using It could be a bug you found It could be someone’s name you need to talk with, or a document you needed to know how the function worked, capture everything.

As you do your testing, you come up with new ideas. List those under test ideas. You will find you conduct your tests and then take a different path than what was earlier intended. Test cases can be listed under each test idea.

What are test ideas?

Test plan would contain functions, conditions, or the things we want to test but not test ideas. Test ideas are at a lower level of granularity, or more detail. The next layer down would be test cases. You also can use mind maps to record and above sections would be the main areas of the map. Remember that without some rigour around testing, you will be performing ad-hoc testing not ET.

Practical component of the session…

Mission for the day was to “Find the worst possible calculation from an online parking calculation form, http://www.grr.org/ParkCalc.php”

At the end of the given time, one of the participants shared below charter document.

  • Findings :
    01/01/-292276999999 31/12/292276999999 $ 5,124,095,144,284,634.00 (2.13503964345E+14 Days, 0 Hours, 0 Minutes) 01/01/-999999999999999999999999999999999999999999999999999999 01/01/-999999999999999999999999999999999999999999999999999999 01/01/-999999999999999999999999999999999999999999999999999999 $ 24.00
    (0 Days, 23 Hours, 59 Minutes)
  • Incorrect date (February 31) works
  • Incorrect formatting (31A/12B/2016) works
  • Incorrect formatting with a typo (3A1/12B/2016) returns 3/12/2016
  • Negative dates works but after a certain point they return incorrect numbers.
    Dates too high produce errors.
    Incomplete entry date field (01/01) starts at 2000.
    Empty fields produce errors

The thought process for the above report is described as below

  • Test actual dates first and see the results
  • Test dates before current time
  • Test whether negative dates work
  • Test whether dates that aren’t formatted properly (MM/DD/YYYY) work (so more than 2/2/4 digits)
  • Test whether dates that aren’t real dates work
  • Test the highest and lowest possible numbers without errors (within reason, I didn’t get the exact numbers)
  • Test what happens when letters are used
  • Test what happens when fields are incorrectly filled in (no year) or simply left blank.

Examples charter document for the above exercise.

Data Date and Time entry
Information sources
  • Information on the page
  • Another page that helps to identify what is important for the website & company
  • Another website with a parking calculator that you are comparing your testing results to
  • Website standards Accessibility guidelines
Procedure Random Testing Procedure
  • To find the worst parking calculation which encompasses high cost
  • Negative cost
  • Invalid cost
  • Zero cost
  • Happy path
  • Negative scenarios
Test ideas
  • second diff
  • Day diff
  • Month diff
  • Year diff
  • Extreme diff
  • Incorrect calculation on valet parking (it should be $12 for 5 hr or less), but currently it is $12 for less than 6 hr
  • When “Leaving time” is in past , there is no error message displayed (example 3 am to 12 am).

Charter is to explain to yourself how you did the main test procedure. It should be written with as much or little detail as required to help you do the tests again, or for others to see how you did them.

As we see ET is very difficult to do well. But with a loose template that is made up of silos or containers, it can be very fulfilling to a good tester. The basic template above is just a guideline. Filling information to those sections without customizing for your project. When you are investigating a product, you are trying to prove or disprove your mission statement. Everything you do is going towards that outcome. ET is not suitable if you want to do “just some testing”. You need to be focussed and organised to perform ET.

What happen to the charter document after the test session?
Charter document can be used in future testing on the same area and you can see how the tests setup, coverage, ideas to be testes and the questions unanswered along with summary of findings.

So what does doing ET right look like?

  • You are simultaneously running tests
  • Coming up with new test ideas
  • Expanding your test ideas into test cases
  • Writing down questions about things that look odd/interesting/need clarification
    Reading your info sources
  • Noting bugs
  • Formulating your findings summary

Q & A time…

Q: How much time you spend doing any form of exploratory testing?

Rich: That is a really good question. Let me clarify some assumptions. Let’s assume you mean that you, as a tester, has been given a function to test. And you are wondering if you decide how long to test for. So what would happen if you found some serious bugs, and you had a list of 20 test ideas that were still unexplored? What would you say to your lead/manager? You would tell him just that, wouldn’t you? And they might say, ‘gosh, you better keep testing then’. Or they might say, ‘we don’t have time to test anymore’ What would you do then? I suppose that depends on what’s expected to happen at that point. Yes, it would. Or perhaps, you are doing scripted testing for a function. And you want to do some exploratory as well. Maybe that is part of your question? Um, it was partially just how do you personally decide how much time to sink into any one aspect of exploratory testing. So, for example, if you want to test multiple functions of something within an allotted time – like if you wanted to test the upper bounds of cost in that form, but you also wanted to test for other simpler errors or inconsistencies

Q: Do you test until you find it, or do you just spend a little time doing it and move on after an allotted time even with incomplete answers?

Rich: I suppose this was more from a question on your personal method rather than what’s expected of you from someone else. I would approach this exercise this way:
1. Get a clear understanding of the mission
2. Familiarise myself with the form, and its fields
3. Start to note down test ideas, and test cases under test ideas
4. List out any questions about anything Now, that’s for today’s mission.

But if you mission was more open, like ‘test this form’… Then we do different testing. The focus is different. The focus is less about a bug hunt, and more about coverage. So my test ideas would be less granular, and less detailed. For example,

  • Blank values
  • Boundary tests of fields
  • Double entry form submit
  • Special chars
  • Error conditions
  • Browser coverage
  • Localisation
  • Character sets

In fact, I would approach this problem differently by using a heuristic to help me explore the different parts to the function. I would use FDITOPS. Each letter help me think about the parts.

Functionality – covered above
Data – what datasets can I use including different types of dates, and number types
Integrations – Interfaces – just a basic form submission interface here… but if I inspect the code I might find others
Platform – browsers, multi tabs, sessions, tablet, smartphone Structure – understanding the code and its interaction with the browser So you can see the exploratory test session would uncover a lot of detail about the function.
Operations – who will use it? Where and how will they use it? What will they use it for? Are there certain things that users are more likely to do? Is there user data we could get to help make the tests more realistic?
Time – leap year handling, using form at midnight, past Operation – using form as different personas – business man, holidaying family, fraudulent use

You would need many sessions to cover all the ideas. But you need to negotiate with your lead/manager how many session you need.

FDITOPS is a rearrangement of the keyword heuristic SFDIPOT. SFDIPOT is a James Bach idea. why do you rearrange it? Because SFDIPOT starts with Structure. That’s the least used part of the set of ideas. Its better to start with F, as that’s what everyone thinks of first, and once you get that out of the way, the rest make you think much harder. so if given a set amount of time, would you prioritise being precise and exact with what you can uncover (with, for example, boundaries), or would you try to test the most possible things you can, or would it entirely depend on what’s expected of you? again, more a “how do you personally approach this sort of thing” question.

Q: How to prioritize your testing ?

Rich: I would go after information that helps me to focus my tests, or order them. Once you have a list of test ideas, you need to be selective in which ones you choose. You need to use product risk data, claims, business case, previous bugs, user information… anything… to help you to do this. You can also ask the business users, or the product owner to help you do this ordering. Or they can review your ordering. This means you can start with a good test idea that is worth finding out about, and you cover the ideas that are most valuable. Often, you are left with ideas untested.

Last but not least…

ET is a highly disciplined activity that need practice, and understanding of focus of your testing, reasons for doing it, and collaborating with stakeholders to make sure you are adding valuable testing.