With Sydney testers I got the chance to get involve on a practical session on Exploratory testing (ET). Widely known and used but still a question to most.
Host: Rich Robinson from Sydney Testers meet-up group
Following is a summary of what’s being discussed. .
What is Exploratory testing? A definition.
“Exploratory software testing is a powerful and fun approach to testing yet widely misunderstood. In some situations, it can be orders of magnitude more productive than scripted testing. All testers practice some form of exploratory testing, unless they simply don’t create tests at all. ” – JBatch
“Exploratory software testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the value of her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.” – Kaner
ET is about learning the product. We can discover information by testing. We can decide what to tests and we can design the tests. We can execute them and the results help discover new information about the product. It might be good, bad, or neither. This information helps us to decide new tests, and so on.
Furthermore, ET is more about discovering new information by testing, and using your existing knowledge. You can start from knowing nothing, or knowing ‘everything’. It’s not only for tester who do not know the product but for the product expertise as well. Testers who are naturally curious, fast learners, insightful, and really want to find possible bugs will adopt ET easily despite the level of knowledge of the product.
How to apply these concepts practically?
- Focus an area for testing or identify a purpose. Do not opt to test the whole product
- Add sections such as the purpose, the approach, goal to achieve, what the focus should be to the testing framework
- Clearly distinct focus, objective, or mission for your testing
Example : for a login screen, your focus might be, to login as the wrong person.
- Assign small parts of tests to each member
- Time box each session
- Record the session. You always can use a notepad.
This document is referred as a charter document. Sections added to the charter document are referred as containers. These sections can be filled up later. The mind-set is that you are in full discover and info logging mode.
Example for sections
- Information sources : Where do you get information about the item user test
- Data to be used for the tests
- Procedure to follow or available
- Test ideas : The scenarios worth checking out. A backlog of tests that needs to be executed.
- Findings : Overall important information found. The summary of your report
More on filling the sections of the charter document…
It could be a question, or observation. It could be the data you are using It could be a bug you found It could be someone’s name you need to talk with, or a document you needed to know how the function worked, capture everything.
As you do your testing, you come up with new ideas. List those under test ideas. You will find you conduct your tests and then take a different path than what was earlier intended. Test cases can be listed under each test idea.
What are test ideas?
Test plan would contain functions, conditions, or the things we want to test but not test ideas. Test ideas are at a lower level of granularity, or more detail. The next layer down would be test cases. You also can use mind maps to record and above sections would be the main areas of the map. Remember that without some rigour around testing, you will be performing ad-hoc testing not ET.
Practical component of the session…
Mission for the day was to “Find the worst possible calculation from an online parking calculation form, http://www.grr.org/ParkCalc.php”
At the end of the given time, one of the participants shared below charter document.
- Findings :
01/01/-292276999999 31/12/292276999999 $ 5,124,095,144,284,634.00 (2.13503964345E+14 Days, 0 Hours, 0 Minutes) 01/01/-999999999999999999999999999999999999999999999999999999 01/01/-999999999999999999999999999999999999999999999999999999 01/01/-999999999999999999999999999999999999999999999999999999 $ 24.00
(0 Days, 23 Hours, 59 Minutes)
- Incorrect date (February 31) works
- Incorrect formatting (31A/12B/2016) works
- Incorrect formatting with a typo (3A1/12B/2016) returns 3/12/2016
- Negative dates works but after a certain point they return incorrect numbers.
Dates too high produce errors.
Incomplete entry date field (01/01) starts at 2000.
Empty fields produce errors
The thought process for the above report is described as below
- Test actual dates first and see the results
- Test dates before current time
- Test whether negative dates work
- Test whether dates that aren’t formatted properly (MM/DD/YYYY) work (so more than 2/2/4 digits)
- Test whether dates that aren’t real dates work
- Test the highest and lowest possible numbers without errors (within reason, I didn’t get the exact numbers)
- Test what happens when letters are used
- Test what happens when fields are incorrectly filled in (no year) or simply left blank.
Examples charter document for the above exercise.
||Date and Time entry
- Information on the page
- Another page that helps to identify what is important for the website & company
- Another website with a parking calculator that you are comparing your testing results to
- Website standards Accessibility guidelines
||Random Testing Procedure
- To find the worst parking calculation which encompasses high cost
- Negative cost
- Invalid cost
- Zero cost
- Happy path
- Negative scenarios
- second diff
- Day diff
- Month diff
- Year diff
- Extreme diff
- Incorrect calculation on valet parking (it should be $12 for 5 hr or less), but currently it is $12 for less than 6 hr
- When “Leaving time” is in past , there is no error message displayed (example 3 am to 12 am).
Charter is to explain to yourself how you did the main test procedure. It should be written with as much or little detail as required to help you do the tests again, or for others to see how you did them.
As we see ET is very difficult to do well. But with a loose template that is made up of silos or containers, it can be very fulfilling to a good tester. The basic template above is just a guideline. Filling information to those sections without customizing for your project. When you are investigating a product, you are trying to prove or disprove your mission statement. Everything you do is going towards that outcome. ET is not suitable if you want to do “just some testing”. You need to be focussed and organised to perform ET.
What happen to the charter document after the test session?
Charter document can be used in future testing on the same area and you can see how the tests setup, coverage, ideas to be testes and the questions unanswered along with summary of findings.
So what does doing ET right look like?
- You are simultaneously running tests
- Coming up with new test ideas
- Expanding your test ideas into test cases
- Writing down questions about things that look odd/interesting/need clarification
Reading your info sources
- Noting bugs
- Formulating your findings summary
Q & A time…
Q: How much time you spend doing any form of exploratory testing?
Rich: That is a really good question. Let me clarify some assumptions. Let’s assume you mean that you, as a tester, has been given a function to test. And you are wondering if you decide how long to test for. So what would happen if you found some serious bugs, and you had a list of 20 test ideas that were still unexplored? What would you say to your lead/manager? You would tell him just that, wouldn’t you? And they might say, ‘gosh, you better keep testing then’. Or they might say, ‘we don’t have time to test anymore’ What would you do then? I suppose that depends on what’s expected to happen at that point. Yes, it would. Or perhaps, you are doing scripted testing for a function. And you want to do some exploratory as well. Maybe that is part of your question? Um, it was partially just how do you personally decide how much time to sink into any one aspect of exploratory testing. So, for example, if you want to test multiple functions of something within an allotted time – like if you wanted to test the upper bounds of cost in that form, but you also wanted to test for other simpler errors or inconsistencies
Q: Do you test until you find it, or do you just spend a little time doing it and move on after an allotted time even with incomplete answers?
Rich: I suppose this was more from a question on your personal method rather than what’s expected of you from someone else. I would approach this exercise this way:
1. Get a clear understanding of the mission
2. Familiarise myself with the form, and its fields
3. Start to note down test ideas, and test cases under test ideas
4. List out any questions about anything Now, that’s for today’s mission.
But if you mission was more open, like ‘test this form’… Then we do different testing. The focus is different. The focus is less about a bug hunt, and more about coverage. So my test ideas would be less granular, and less detailed. For example,
- Blank values
- Boundary tests of fields
- Double entry form submit
- Special chars
- Error conditions
- Browser coverage
- Character sets
In fact, I would approach this problem differently by using a heuristic to help me explore the different parts to the function. I would use FDITOPS. Each letter help me think about the parts.
Functionality – covered above
Data – what datasets can I use including different types of dates, and number types
Integrations – Interfaces – just a basic form submission interface here… but if I inspect the code I might find others
Platform – browsers, multi tabs, sessions, tablet, smartphone Structure – understanding the code and its interaction with the browser So you can see the exploratory test session would uncover a lot of detail about the function.
Operations – who will use it? Where and how will they use it? What will they use it for? Are there certain things that users are more likely to do? Is there user data we could get to help make the tests more realistic?
Time – leap year handling, using form at midnight, past Operation – using form as different personas – business man, holidaying family, fraudulent use
You would need many sessions to cover all the ideas. But you need to negotiate with your lead/manager how many session you need.
FDITOPS is a rearrangement of the keyword heuristic SFDIPOT. SFDIPOT is a James Bach idea. why do you rearrange it? Because SFDIPOT starts with Structure. That’s the least used part of the set of ideas. Its better to start with F, as that’s what everyone thinks of first, and once you get that out of the way, the rest make you think much harder. so if given a set amount of time, would you prioritise being precise and exact with what you can uncover (with, for example, boundaries), or would you try to test the most possible things you can, or would it entirely depend on what’s expected of you? again, more a “how do you personally approach this sort of thing” question.
Q: How to prioritize your testing ?
Rich: I would go after information that helps me to focus my tests, or order them. Once you have a list of test ideas, you need to be selective in which ones you choose. You need to use product risk data, claims, business case, previous bugs, user information… anything… to help you to do this. You can also ask the business users, or the product owner to help you do this ordering. Or they can review your ordering. This means you can start with a good test idea that is worth finding out about, and you cover the ideas that are most valuable. Often, you are left with ideas untested.
Last but not least…
ET is a highly disciplined activity that need practice, and understanding of focus of your testing, reasons for doing it, and collaborating with stakeholders to make sure you are adding valuable testing.