Monday, July 28, 2014

It's Just Semantics

Why is it so important to say exactly what we mean? Checking vs testing, test cases aren't artefacts, you can't write down a test, best practice vs good practice in context - isn't it a lot of effort for nothing? Who cares? 

You should. If you care about the state of testing as an intellectual craft then you should care. If you want to do good testing then you should care. If you want to be able to call out snake oil salesmen and con artists who cheapen your craft then you should care. If you want to be taken seriously then you should care. If you want to be treated like a professional person and not a fungible data entry temp then you should care.

Brian needs to make sure that their software is okay. The company doesn't want any problems in it. So they hire someone to look at it to find all the problems. Brian does not understand the infinite phase space of the testing problem. Brian wants to know how it's all going - if the money the tester is getting is going somewhere useful. He demands a record of what the tester is doing in the form of test cases because Brian doesn't understand that test cases don't represent testing. He demands pass/fail rates to represent the quality of the software because Brian doesn't understand that pass/fail does not mean problem/no problem. The test cases pile up, and writing down the testing is becoming expensive so Brian gets someone to write automated checks to save time and money because Brian doesn't understand that automated checks aren't the same as executing a test case. So now Brian has a set of speedy, repeatable, expensive automated checks that don't represent some test cases that don't represent software testing and can only pretend to fulfil the original mission of finding important problems. I won't make you sit through what happens in 6 months when the maintenance costs cripple the project and tool vendors get involved.

When we separate "checking" and "testing", it's to enforce a demarcation to prevent dangerous confusion and to act as a heuristic trigger in the mind. When we say "checking" to differentiate it from "testing" we bring up the associations with its limitations. It protects us from pitfalls such as treating automated checks as a replacement for testing. If you agree that such pitfalls are important then you can see the value in this separation of meaning. It's a case of pragmatics - understanding the difference in the context where the difference is important. Getting into the "checking vs testing" habit in your thought and language will help you test smarter, and teach others to test smarter. Arguing over the details (e.g. "can humans really check?") is important so we understand the limitations of the heuristics we're teaching ourselves, so that not only do we understand that testing and checking are different in terms of meaning, but we know the weaknesses of the definitions we use.

So, we argue over the meaning of what we say and what we do for a few reasons. It helps us talk pragmatically about our work, it helps to dispel myths that lead to bad practices, it helps us understand what we do better, it helps us challenge bad thinking when we see it, it helps us shape practical testing into something intellectual, fun and engaging, and it helps to promote an intellectual, fun and engaging image of the craft of testing. It also keeps you from being a bored script jockey treated like a fungible asset quickly off-shored to somewhere cheaper.

So why are people resistant to engaging with these arguments?

It's Only Semantics

When we talk about semantics let's first know the difference between meaning and terminology. The words we use contain explicature and implicature, operate within context and have connotation. 

Let's use this phrase: "She has gotten into bed with many men".

First there's the problem of semantic meaning. Maybe I'm talking about bed, the place where many people traditionally go in order to sleep or have sex, and maybe you think I'm talking about an exclusive nightclub called Bed.

Then there's the problem with linguistic pragmatic meaning (effect of context on meaning). We might disagree on the implication that this women had sex with these men. Maybe it's an extract of a conversation about a woman working with a bed testing team where she's consistently the only female and the resultant effects of male-oriented bed and mattress design.

Then there's the problem with subjective pragmatic meaning. We might agree that she's had sex with men, but might disagree on what "many" is, or what it says about her character, or even agree that I'm saying that she's had sex with many men where I am implying something negative about her character but you disagree because you've known her for years and she's very moral, ethical, charitable, helpful and kind.

So where we communicate we must understand the terms we use (in some way that permits us to transmit information, so when I say "tennis ball" you picture a tennis ball and not a penguin), and we must appreciate the pragmatic context of the term (so when I say "tennis ball" you picture a tennis ball and not a tennis-themed party).

When we say "it's only a semantic problem" we're inferring that the denotation (the literal meaning of the words we use) is not as important as the connotation (the inference we share from the use of the words).

Firstly, to deal with the denotation of our terminology. How do we know that we know what we mean unless we challenge the use of words that seem to mean something completely different? Why should we teach new people to the craft the wrong words? Why do we need to create a secret language of meaning? Why permit weasel words for con artists to leverage?

Secondly, connotation. The words we use convey meaning by their context and associations. Choosing the wrong words can lead to misunderstandings. Understanding and conveying the meaning IS the important thing, but words and phrases and sentences are more than the sum of their literal meaning, and intent and understanding of domain knowledge do not always translate into effective communication. Don't forget that meaning is in the mind of the beholder - an inference based on observation processed by mental models shaped by culture and experience. It's arguments about the use of words that lead to better examination of what we all believe is the case. For example some people say that whatever same-sex marriage is it should be called something else (say, "civil union") because the word marriage conveys religious or traditional meaning that infers that it is between a man and a woman. The legal document issued is still the marriage licence. We interpret the implicature given the context of the use of these words which affects the nature of the debate we have about it. We can argue about "marriage" - the traditional religious union, and we can argue about "marriage" - the relationship embodied in national law and the rights it confers. We can also talk about "marriage" - the practical, day to day problems and benefits and trials and goings on of being married.

Things Will Never Change

Things are changing now. Jerry Weinberg, James Bach, Michael Bolton and many others' work is rife with challenging the status quo, and now we have a growing CDT community with increasing support from people who want to make the testing industry respected and useful. There's work in India to change the face of the testing industry (Pradeep Soundararajan's name comes to mind). There's STC, MoT, AST and ISST. The idea that a problem won't be resolved is a very poor excuse not to be part of moving towards a solution. Things are always changing. We are at the forefront of a new age of testing, mining the coalface of knowledge for new insight! It's an incredibly exciting time to be in a changing industry. Make sure it changes for the better.

It's The Way It's Always Been

Yes, that's the problem.

I'll Just Deal With Other Challenges

Also known as "someone else will sort that out". Fine, but what about your own understanding? If you don't know WHY there's a testing vs checking problem then you're blinding yourself to the problems behind factory testing. If you don't investigate the testing vs checking problem then you won't know why the difference between the two is important to the nature of any solution to the testing problem, and you're leaving yourself open to be hoodwinked by liars. Understand the value of what you're doing, and make sure that you're not doing something harmful. At the very least keep yourself up to date with these debates and controversies.

There's another point to be made here about moral duty in regard to such debates and controversies. If you don't challenge bad thinking then you are permitting bad thinking to permeate your industry. If we don't stand up to those who want to sell us testing solutions that don't work then they will continue to do so with complete repudiation. I think that this moral duty goes hand-in-hand with a passion for the craft of testing, just as challenging racist or sexist remarks goes hand-in-hand with an interest in the betterment of society as we each see it. It's okay to say "I'm not going to get into that", but if you claim to support the betterment of the testing industry and its reputation as an intellectual, skilled craft then we could really use your voice when someone suggests something ridiculous, and I'd love to see more, innovative approaches to challenging bad or lazy thinking.

"Bad men need nothing more to compass their ends, than that good men should look on and do nothing." - John Stuart Mill

Sounds Like A Lot Of Effort

Testing is an enormously fun, engaging, intellectual activity, but it requires a lot of effort. Life is the same way. For both of these things what you get out of it depends on what you put in.

Arguments Are Just Making Things Less Clear/Simple

This is an stance I simply do not understand. The whole point of debate is to bash out the ideas and let reason run free. Clear definitions give us a starting point for debate, debate gives us rigour to our ideas. I fail to see how good argument doesn't make things better in any field.

As for simplicity, KISS is a heuristic that fails when it comes to the progress of human knowledge. We are concerned here with the forefront of knowledge and understanding, not implementing a process. Why are we so eager to keep it so simple at the expense of it being correct? I've talked to many testers and I've rarely thought that they couldn't handle the complexity of testing, as much as they could the complexity of life. I don't know who we're protecting with the idea that we must keep things simple, especially when the cost is in knowledge, understanding and the progression of our industry.

I've been reluctant to release this post. It represents my current thinking (I think!), but I think it's uninformed which may mean that it's not even-handed. Linguistics and semantics are fairly new to me. I'm reading "Tacit and Explicit Knowledge" at the moment, I may come back and refine my ideas. Please still argue with me if you think I'm wrong, I have a lot to learn here.

Friday, July 11, 2014

Best Practices Aren't

Best Practices Aren't

I assume that I'm directing myself at a group of people who know that there's no such thing as a Best Practice, and that we shouldn't use the term. There's a bit of consternation around this, some people defending the term whilst understanding the nature of contextual value of practices.

I'm working under the assumption that you've read this:

Of course there are no Best Practices! Of course practices only apply to a particular context! Okay, with that under our belt I'd like to talk about why we shouldn't use the term at all, and why it matters that we shouldn't.

It's A Trap!

Saying what we mean is important. We can call poison "food" and assume that tacitly we all know it's actually poison, really - until the amateur or black-and-white thinker wanders in and eats it, or some dishonest salesperson sells it to someone as food (after all, that's what we're calling it). In the same way we can all agree that "best" just means a particularly good thing to do in the context, but when people use "best practices" as a benchmark, ignoring the subtleties in the change in context, bad things can happen. Someone can look at a standard in one company and assume it will work for them because they're in the same industry - but there's a lot more to context than that. There's methodologies used, hours worked, company culture, relationships between people and teams, budget, development speed, even the geographical location of the office building. So why do we need this secret society of people who understand that "Best Practice" doesn't really mean "best"? Especially when we can talk about practices without using that term in the first place? Maybe it's some sort of egotistical elitism or some desire to find tranquil simplicity in a complex world, but either way it should affront your professional ethics to want to be part of the Secret Society of Avoidable Confusion.

It's Lazy Thinking

Understanding the flaws in a practice - the contexts in which it won't work or has little value - is important. Knowing what we should do is better than assuming that emulating someone else's practices will be just fine. It's thinking like this that permits tool vendors to sell snake oil cure-alls for the testing problem, and permits the persistence of worthless certifications. We can say "of course it's for a particular context" but that's because we don't want to have to think about the effect of context on the value of our practices because then we'd have to do some work. If you're the sort of person that believes that "automation" is more than an automatic check execution system to replace good, exploratory testing then you understand the importance that context has on the value of a practice. So don't let your brain go on holiday just because someone stuck the word "best" in front of "practice" and claimed it as some sort of benchmark of greatness, even if they permit you the luxury of implicit contextual value. Think critically, and don't let anyone stop you from doing so.

It Replaces Discussion

The "of course it's for a particular context" argument is the end of the discussion of our practices when we hold Best Practices up as a benchmark. Thinking about our practices, and their value, with the influence of practices we know work well elsewhere is not something to avoid or be ashamed of. There seems to be some resistance to the idea of eliminating best practices because of laziness, and some fear of "reinventing the wheel" - except that we're not re-inventing a wheel, we're adapting the wheel, changing the wheel, assigning cost to the wheel, using a different kind of wheel, examining how suitable the wheel is for our purpose and perhaps deciding that we don't need a wheel. "Don't re-invent the wheel" does not mean that we should use stone cylinders with a hole in them on our motor vehicles. Use the right wheel for what you need to do, even if someone says that a particular wheel is the "Best". Use your awesome tester powers of critical thinking.

It's Not What "Best" Means

"Best" is a superlative. The form of the adjective "good" that infers that it is the greatest of any other possible degree of something. When you say (without hyperbole) that "this is the best-tasting sandwich I've ever eaten" you are saying that there is no sandwich that you have ever eaten that tastes better than this one. Best Practice simply does not make sense, even with context factored in. Even for a particular context there is no Best Practice that we can know is the best one. "Best Practice for this context" means "There is no practice that exists or could be thought of that could be better than this one for this context" - which is quite the claim! Don't let people get away with that sort of thing.

An Alternative

If you really need a short-handed way to say "Best Practice" but actually mean something more like "good practice for a particular context" here are some ideas:
  • (I've found it to be an) Effective Practice
  • Cool Idea
  • The "<your name here>" Method
  • (My) Favourite Practice (for)

Or alternatively you could just talk openly and honestly about practices, where they seem to work for you, and where and how they could be applied to good affect.

Wednesday, July 2, 2014

STP Podcast - Automation In Testing with Richard Bradshaw

Listen here:

This was a good listen! So good I'm inspired to comment in more space than can fit in a tweet. I do agree with basically everything mentioned, which is a shame, but it did bring up ideas for me that I hadn't considered and ways of expressing things I hadn't heard yet, which is a treat.

Forgive the quality, I rushed the whole post.

Automation Is A Tool

Richard is clear and passionate on this topic, and I could not agree with him more. Automation is a tool, and as a tool it has to serve a test strategy in some way. Automation is so often seen as chasing some benefit that doesn't exist. It's like buying a new battery-powered electronic drill purely under the assumption that it will save you time with your DIY. Well it depends on what you're going to use it for - why do you need a drill, and why does it need to be this drill? A drill doesn't mean you don't have to drill holes any more. It also requires maintenance. It requires skill to use. It can enhance your ability to drill holes, and it can enhance your ability to make mistakes and do damage. It has disadvantages over other hole-boring tools for certain tasks. It requires exploratory learning during its use, examining and interpreting the results.

Exploratory Automation

I agree with the statements in the podcast. I'd go as far as to say that there's no such thing as an exploratory tool. I'd accept "pseudo-exploratory", maybe. Exploration requires learning, and only humans can learn in that way. For a tool we have to encode how it interacts with the product, what oracles it can use and how to use them, how to make decisions... and we cannot encode it to cope with the implicature of that information nor can we code it to be inspired to do anything we haven't yet thought of. Humans can notice a symptom, follow it, design new tests, learn from the results, design further tests, and so on leveraging heuristic principles to find problems or build better mental models. They can get inspired by results and use that to explore new ideas based on their understanding of risk (heuristics such as "it's bad when people get hurt" that are hard to teach to a machine).


"How do I measure my manual testing and have visibility into the value of my manual testers?"
"How do I have visibility into the value of my automation?"

Richard gave a great answer to this.

There is no way to usefully quantitatively measure the value (someone's regard of the importance or usefulness of something) of a tool. That's the problem, it's subjective. Don't automate to save money. Automate when it serves the test strategy. Are you willing to invest in the quality and speed of the information  testing provides, or are you looking to replace an expensive but necessary service? You wouldn't buy a new IDE for your development team with fancy code tools so that you could fire some of your developers and get return on investment, so you shouldn't do it with testing (unless your testers are THAT replaceable, in which case your problems can't be solved with a tool). As Richard says, "Automation is there to serve the team".

I'm always curious as to why people want to value their tools like that. I understand the hesitation if someone is buying an automated check execution tool under impossible promises from a vendor, but I'm more interested in reality. Let me ask this: How do you get visibility into the value of your car? Your car (or a car you might buy if you don't have one) is a tool that you can use. Well, you can look at the cost of the car, maintenance and fuel but you have to measure it against how useful it is. How do you measure how useful something is? It's a subjective matter. What do you use your car for? Could you do without a car? What's the difference in utility between having the car and not having the car? What are the drawbacks? How do you factor those into the utility? What about things you haven't thought of?


I also really liked Richard's point around here of "automation is dumb" and that it doesn't need to be intelligent because we have intelligent testers to use it.

This wasn't even half of the content of the podcast, so I'd recommend listening to the whole thing. They go on to discuss automation tools beyond automatic check execution and how to use them to inform mental models, the kinds of people who do different types of testing, reporting and so much more. I'd like to write about this too but I only have so much time!

Thanks to STP, Mark Tomlinson and Richard Bradshaw for a great podcast!