Looking for Test Tools?

Are you looking for tools to help you in your testing?

You are probably in one of two situations.

Situation One

You have a problem for which you hope there is a tool to make your life easier. This is usual, and fine, and I wish you well in your journey. The way I usually do this is to identify the problem I have, and search Google for potential solutions, and you're probably not even reading this.

Situation Two

You are perusing a page of tools used by, and suggested by, other people. There is nothing intrinsically wrong with this, but here is where you need to be wary, and there are two main reasons that I believe you should:

Your context is not their context

What works for one person doesn't work for all others. This is fairly obvious, but you shouldn't confuse the process of testing with the tools of testing - if you try to save effort by changing the way you work to fit in with a tool then you're allowing the tool to potentially reduce how well you can work. You need to temper your lust for "smarter working" with a stoic judgement of the benefit the tool is going to provide.

Tools don't do any work

If I went out and bought a trowel and bricks and cement mixer and so on all I would end up with is well-prepared ingredients for a brick construction, because I do not know how to lay bricks. Anything I made would be slanted, unbalanced, weak and possibly dangerous. The tools we have to help us perform tasks are not replacements for a complete job but a short-cut to save time, effort or frustration in reaching our goal. If you seek, or use, a tool to try to do your testing for you ensure that you are comfortable with the fact that you are no longer testing. When you use a tool to help you you are taking yourself out of that part of the equation. This, on balance, may be an acceptable loss given the constraints on your time or sanity, or the lack of need for your personal touch for the task, but it is a loss nonetheless.


So, in conclusion, if you want to use a tool to help you do your testing then you need to balance the negative with the positive. The tasks you are replacing, your newly found lack of attention, your assumption the tool is doing the job, if the tool will work for your context, the maintenance costs and so on versus the time, effort and frustration you will save by using it. Don't confuse tools with processes.

Exploratory Testing, Management and Tools

Disclaimer: You wouldn't eat with a hammer and you wouldn't bang a nail in with a fork. Tools are options. They are a bag of tricks you keep by your side. A mechanic doesn't use the whole tool chest to check your breaks, and you shouldn't let software tools be your dogmatic methodology. Stay light, simple and efficient is my advice. Your own set of tools will be dependent on your own context. But if you want some more brushes in your paintpot here is a list of some exploratory and session-based management tools:

Second Disclaimer:

You will find a better list here: http://www.ministryoftesting.com/resources/software-testing-tools/

Exploratory Testing & Management Tools

Note Taking

Paper and Pen

Paper & pen is flexible (in both senses), versatile, expressive, cheap, easily accessed, and useful. It stores well in a suitable filing system, it's portable, and generally ubiquitous. Plus it doesn't need a computer, or even electrical mains power or batteries to use. Not to be ignored.

Notepad

A limited version of paper & pen for the computer! Your OS probably has an equivalent.

Rapid Reporter

Most ET practicioners have heard of this, in my experience. It is a "note taking application that aids in the reporting process of SBTM". It doesn't take up much real-estate, it sits always-on-top, and has a transparency slider, so it's very convenient to use. It's standalone (no installation), and outputs to the easy-to-parse CSV format. It's customisable, easy to control with the keyboard, and does its job very well, I highly recommend it.

http://testing.gershon.info/reporter/

Session Capture

Video Screen Recorders

Screenr, Camtasia, Hypercam

They all record everything you do, and if the disk space these videos use isn't an issue for you then they are a valuable asset. Once you have the video at hand you can answer those questions... Did it do that before? Did I click the button before or after? Why did it work the second time and not the first?...

qTrace

After a couple of suggestions I downloaded qTrace. I've played only marginally with this so far but it's remarkably good. You boot it up, select the application to record, hit record, then start testing. When you hit stop it presents you with every action you performed, and these trace files can be saved. You can even specify the number of screens you want kept so that you can raise the processes as a bug, and the software interacts directly with many tracking systems including JIRA (ours).. although you need to pay a small fee for this (and to remove a small ad). It also saves a very useful table of environment information (OS, browser version, CPU, RAM, Available RAM, current user...).
It has facilities to force a screen capture, add a notes to the steps, blur out sensitive information on the page, add arrows/boxes/text to your screens, and more.
Together with good notes I consider this a very viable candidate for our ET sessions. One thing it doesn't seem to do is track timestamps, so these would have to go in your notes.

Other Useful Tools

Mindmap Tools

FreeMind, XMind, MindMeister

JIRA Plugin: Bonfire [£]

A good-looking JIRA plugin that sits in your browser and handles private or public test sessions, bug report templates and more. Worth a look if you're planning on using session-based management.



Any further suggestions please tweet me @kinofrost, letting me know if you'd like direct credit for it (defaults to "no" for reasons of privacy and spam).

Many thanks to the many testers of Twitter, and a very useful RT from Software Testing Club for the suggestions.

Zendo: Testing in game form

I love Zendo. Its fans will often describe it as the best game, or gaming concept, ever made. It is deeply strategic without being overly competitive, in my opinion, and is also a lot of fun, especially with good players.

Here it is on Wikipedia:
http://en.wikipedia.org/wiki/Zendo_(game)

I suggest getting a few playing pyramids and a copy of Playing With Pyramids from Looney Labs.

Took me far too long (plus the time to watch a James Bach lecture, plus the time for it to sink in) to see that it contains a lot of the same thought processes as testing. My prediction: a good exploratory tester would kick arse at Zendo.

Here's the idea behind Zendo (icehouse piece version).

First you have a set of "pyramids" (icehouse pieces, treehouse pieces, or whatever they're calling them these days). They are often transparent, coloured, stackable pieces that come in sets of 15 (5 each of small, medium and large, or 1-point, 2-point and 3-point pieces if you prefer to call them that).

You also need a pile of black stones and a pile of white stones.

The "Master" player thinks up a rule, creates two "Koans" (small constructions made out of the pieces), one of which obeys the rule and one which does not. A rule could be "A koan has the Buddha nature if and only if it contains a blue piece" or "A koan has the Buddha nature if and only if it contains two medium pieces" or "...contains a blue piece pointing at a red piece" or "the value of the pieces is a square number"...etc.

Basically the idea for the other players is to guess this rule by building koans and having them marked as having the rule (white stone) or not (black stone) until they can determine enough about the system to guess at the rule.

If the master cannot build a koan that disproves the guess (build one that has the Buddha nature but disobeys the guessed rule, or build one that does not have the Buddah nature and obeys the guessed rule) then that player wins.

In order to do this you need to evaluate what the white koans have that the black koans do not and vice versa. Each guess will form part of the rest of your test plan as the resulting information is useful feedback. You need to make "quick attack" guesses that halve the problem or find useful information quickly rather than slow, plodding ones or wild guesses that make little advance to your current knowledge. You must not repeat tests (build exact copies of other koans) and reduce partial repetition to a minimum when you have enough knowledge to know better (building near-exact copies of other koans, such as the same koan with different colours after you have ruled out colour being important in the rule).

If it's not a good similie for exploratory testing I fail to see how.

Just a quick thought, I might formulate it into a more formal assessment later.

QA & Testing Resources

I'm investigating the world of QA & Testing Blogs and resources!

Mine has its moments of testing-related goings on in between robotics, ranting and philosophical silence, but if that simply isn't enough there are many more resources available. I will try to update this list based on comments, and things that I find in my search or buried in my bookmarks. Some I know well, some I do not, and the list will require a tidyup. If I particularly enjoy or find use in a resource I might give it a little star or something, who knows? Nobody, that's who.

If you would like your site removed from this list please contact me or leave a comment and I will do so. If you'd like it added please leave a comment and I'll give it a once over and add it to the list!

Blogs

Phil's Expected Results
Andy Glover's http://cartoontester.blogspot.co.uk/
QA Hates You
Trish Khoo's testing blog
Testy Redhead
Alister Scott's Watirmelon [Watir/Watir-webdriver]
Hugh McGowan's The Science of Testing
Test Side Story
Michael Bolton's blog
James Bach's blog

Videos

http://www.satisfice.com/videos.shtml
MoT Test Bash

Groups / Communities / Forums

Software Testing Club
Ministry of Testing
Developer Testing
Test Driven
SQA Forums
Sticky Minds

Tools

Podcasts

Test Metrics - Dangers In Communication

At the Software Testing Club meet last night many interesting points were made but one that captured my interest in particular was testing metrics.

What Are Metrics?

Testing, like most things, is full of numbers and statistics. Numbers and statistics are everywhere. And metrics are just numbers that have reason to them. Metrics are "an analytical measurement intended to quantify the state of a system"¹. 


Unfortunately numbers say whatever you would like them to say. I always return to an old adage taught to me at school:

Information = Data + Meaning

So the numbers in statistics are data, and what they refer to is the meaning. Or the numbers are the quantification and what they refer to is the state of the system. But this is all that these numbers refer to, and that is the inherent problem with metrics.

Why Do We Use Metrics?

Well to "quantify the state of the system" of course! But is that really why we use metrics? We know that's what they refer to but metrics have a habit of miscommunication meaning and I think the operative word we hit on last night was context.


Managers love metrics. They like reports, charts and graphs because they can gain information about a system that they know perhaps very little about or don't have time to understand. It's easier to go to your boss with "Our bug count is 17% lower this quarter" than "well, looks like it's going okay and the testing guys say that the quality looks higher". But do the numbers tell the truth about the system? If making you or your team look good, or putting your head down and getting away with nominal improvement is your goal then metrics are a gift from heaven.... but if relaying the true quality of your product is the goal then qualification and context are vital.


Our discussion last night prompted the example of "severity". Saying "We have 20% fewer Severity One bugs" sounds great, but this statement should prompt a lot of questions, such as:

• What on earth is a "Severity One" bug?
• What does it mean to 'have' the bug? That we found fewer? That we fixed more than we found?
• Is 'having' fewer of these bugs really a good thing? Are we spending time fixing bugs and not finding them? Are we simply not finding them any more when we should?

Science!

Science is a methodology for finding truth about the universe despite our own cognitive biases. One look at an optical illusion shows us that seeing isn't believing, and starting with an answer and then trying to prove it makes the likelihood of experimental data showing the pre-selected answer more likely ("hmmn, that doesn't look right, I'll take that reading again..."). We are pattern seeking animals, we tend to confirm our own beliefs and look for "hits" and forget the "misses" - this is confirmation bias. It's why we see shapes in the clouds and the face of Jesus on a crisp, we are evolutionarily programmed to respond to patterns and false negatives (is that rustling in the trees wind or an enemy? Better run just in case).


All this means that we should not interpret the data after deciding what we want it to say, otherwise we are creating data that does not tell the truth. They are the most basic rules of scientific philosophy. It might make us look good or cover up our mistakes or those of our colleagues, but it's not the genuine state of the world, or the system under test.

How Should We Use Metrics?

Interpret the numbers on behalf of those receiving them, do not let them do it themselves. Those collecting the metrics are probably at the best point of understanding, so add the context and qualifications there before handing them off to someone else. Do not use confusing short-hand terms like "code coverage" and "bug count" without explanation. Formalise the meaning of domain terms like "severity" so that testers can categorise issues by a domain-wide understanding of the term.

This all feeds into our need to understand more complex things about the system than "number of bugs fixed" or "test coverage", to the ability to understand opportunity cost and apply it to schedules, risk assessment, functional prioritisation and so much more. 

 ¹ http://en.wikipedia.org/wiki/Metric

Robotics ho!

To the land of robotics!

I am going to build my first Arduino line-following robot. It is decided.

I have a shopping list, I have the tools, I may be moving house within 2 weeks so I can't have anything delivered yet, but I will start the planning process and decide where I can get a cheap chassis and some cheap wheels from... it may be time to get back to the boot sale! I can't wait to stop being ill so I can chase these things.

If you're following along at home I will need:

* LEDs
* A battery
* A chassis
* 2 hobby servo motors (which I will adapt to run continuously. I have to learn sometime.)
* Nuts/bolts
* Resistors
* Matrix board
* IR Emitter/detectors
* A motor shield
* A caster wheel for the front

I'll try to update with photos when I actually build this thing.

Organising My Life 2

Well I've tried Astrid and Toodledo and Lifetick and a host of other motivational goal-based GTD to-do list widgets and I've come to no satisfactory conclusion on them. I have found out as much what I want as what I didn't want, and I've decided what I want is three things:

1. A to-do list that is simple, has multiple lists, and contains actionable short-term things that need doing.

2. A system for maintaining personal and work projects

3. Something to hold user stories

Number one is still under consideration but at the moment I am using Wunderlist. It is simple and it works.

Number two I stumbled upon via Wunderlist, their new Wunderkit which (even without the social aspect) is a great way of braindumping my larger-scale personal projects. It holds a few points over the dishevelled notes I was holding in OneNote and Google Docs, and has a much more personable layout than both, as well as Evernote which I can't really abide.

Number three, without a doubt, is PivotalTracker which is nothing short of an amazing piece of programming, but I'm always up for suggestions!

My trouble will now be checking all these things each day! When I get home in the evening I will often not remember to check to-do lists. Wunderkit has tasks, but they don't interrupt me enough to ensure they'll get done.

So:

If I have something that needs doing I will put it on Wunderlist. The downsides are no alarms and a slightly fiddly Android app.

If I have a long-term goal or project with subtasks or recurring tasks I will use Wunderkit. The downsides are no alarms, and no visible progress towards the goal, but I like the idea and if I had other people working on a project it does make for a good project collaboration space (although I haven't needed to try others).

And I will continue to use PivotalTracker :).