Move to Wordpress

Hello!

If you're the sort of person who enjoys peaceful silence then you can continue to follow my Cage-esque experiment in self-publishing over at https://secondsignofmadness.wordpress.com/.

A Partial Defence of "Apathy"

Hello!

I have had one reply that I know of to my Apathy post containing useful critique. You can find it here written by a name you may know, Matt Heusser! First I have to say that I'm grateful to have a considered response. Now I'll expand a little on the points raised, and hopefully draw an interesting conclusion.

When you critique someone in public, there is an audience. That observer effect, to borrow a phrase, changes the nature of the conversation.

This is a useful insight, and it shows how dangerous my post could be. I don't know who's reading, and how they might employ the idea that they should shrug off apathy and argue the point - maybe I'm empowering people with immoral aims or ones who misunderstood my intent! I didn't expect anyone to question me on it. Now they have, and I have to pose some clarifications and try to be honest with what I learned and where I was wrong.

Sometimes, it can be tempting to play to the audience. If you think you’re talking to the other person, and they are playing to the audience, you’ll see a bunch of bizarre behaviors that don’t seem to make sense and you won’t be able to figure out from your position.

Wait, did I just play to the audience? Of course I did! But why? It's quite simple: I want to express and emphasise the idea that critique can be constructive for both parties and doesn't have to be a social nightmare or just to further personal ends. We can place the pursuit of further learning and conversational exploration above our need to seem clever... but it does depend on a hidden intent. You have to believe that my intent is to make us more honest with things that matter in our industry, although if you were following my advice you'd be slightly suspicious of my intent and make up your own mind.

So how do I salvage the point? There's not much I can do about intent. What I can do is say that with a critical mind you'll always be better poised to find a more objective truth than without criticism. Just look at scientific progress. I can also say that the critical tools you use in discussions about a subject are ones you can use in testing software.

Instead of “yes, but…” say “yes, and.” Add, ways to do things, or think about that. “Here’s what has worked for me”. Consider the environment.

This is another critical point, the partial answer to which I hid in a footnote. People are emotional creatures and no amount of intent to find the truth will help you convince someone who thinks that your intention is to attack them instead of seeking a better idea or the limitations of one. Of course some people abuse this and deliberately overreact with anger or hurt feelings to make you look cruel, which can be a smokescreen to hide their bad ideas. It's not as simple as I made it seem, far from it.

That's not to say I regret typing it, though. I'd rather people asked the questions (nicely, generally speaking) and discovered what the reaction is. I'd rather people reading publicly-posted advice had the opportunity to look deeper into the subject. If the person posting the advice refuses to defend the advice (no matter the reason) then we can make a useful judgement about how valid that advice might be, although not necessarily how invalid it might be. If the person posting the advice cannot defend the advice then we can make a similar judgement. It's not that the advice is intrinsically bad. Even if you're right that doesn't necessarily make their point wrong, and making them seem wrong also doesn't make their point wrong. But it does mean that you, and anyone else reading it, is subject to a reminder that they should question what they are reading. Maybe someone else could take up their cause.. happens to me all the time and I'm often glad of it.

Consider getting to know the speaker. Find out if feedback is warranted. Ask if feedback is wanted. Ask for the best way to give feedback. [...] Sometimes, the right choice is to say nothing.

If we're talking about feedback this is great advice. If we're talking about tackling bad ideas and keeping us honest it's... a reasonable heuristic. Part of my intent is to get the professional testing community to tactfully request reasons. In most scenarios it doesn't have to look like feedback and it doesn't have to look like critique. When someone says "I had a great time at the testing meetup" you can say "sounds great! Did you learn anything good?". When someone says "I thought that lecture on testing was really good" you can say "sounds like you enjoyed it! Do you think it will change the way you test or think about testing?". When someone says "Bob's a really awful tester" you can say "what did you see that makes you think that?".

There are a lot of little cognitive tools and heuristics we can use to do this (I'm writing something on
some of them now). That last one I used is called the data question [1].

That's a great start. It's positive. It can often be just asking for clarification so that you're on better ground to ask more questions, even if those questions are of yourself! We must create somewhat safe spaces in which to find better solutions, whilst not, in our friendly way, opening ourselves up to nonsense and 

If the idea is bad, if it is actively doing harm to the community, let’s separate that from “you are a bad person.” Character attacks might not be fit for public consumption, but you can attack bad ideas, often by the consequences of those ideas.

Please, let's! Character attacks aren't usually very useful to prove a point.. unless the point you're arguing is about their character.  Making character attacks to prove a point not connected to a character is not going to help you express the idea that you're a professional skeptic (to other professional skeptics) because every professional skeptic should know about the misuse of ad hominem attacks.

Do it too much, though, and things can change; you’ll find your reputation is built on criticism. That can become a very dangerous business. Better to be known for what you stand for.

It's true, it can get to be excessive. It's also very tiring, and can be emotionally challenging, and we need to know that it's probably the same or possibly worse for others. It's possible to lose sight of the learning and focus on making an argument. If you've ever seen two people argue who don't listen to each other you'll have seen how it can get. If you've ever seen an argument where one or more parties have no intention of ever changing their opinion you'll know how pointless it seems. But it does show something important about that party - if they aren't willing to defend their idea or consider that they might be wrong or be open to learning something new then how reliable are they with such matters? That doesn't mean that they're bad people, of course, just that they might be a marketer or salesperson rather than a professional skeptic. Perhaps it's not their job to seek truth in confusion. We can judge the value of their advice for ourselves when we know more about then.

However, I do want to be known as a professional skeptic. I also want to be part of a community that keeps me to account if I say something confusing or disagreeable, so I want to be known as someone who is willing to critique and be critiqued, preferably in a somewhat humanitarian and professional way.

I'd like to invoke the words of someone else to help me here.

I only demand that my colleagues debate with me if they want to continue to be my colleagues. Otherwise, they can do as they like.

— James Marcus Bach (@jamesmarcusbach) February 2, 2016

I assume that my colleagues hold me to this standard: I can defend the ideas I promote. I assume the same goes for all adult professionals.

— James Marcus Bach (@jamesmarcusbach) February 2, 2016

Thanks Kino. If I had to critque, I might say the post was a little naive. That’s cool tho, man.

This is useful to mention, because it was naive. I did not construct a particularly well-formed argument in my post. I did not expand on the details or predict possible important counterarguments. I think my post was important, and served a message, and promoted thought, but it didn't question itself and it didn't urge caution or handle its own weaknesses. Moreover, I knew that I was doing it, which is why I put any defensive detail in a footnote. Mr Heusser spotted this and called me out on it.

Now look where we are. How did I find these additional insights? How is it that we dug deeper on this issue? Because one (of a very select few) took it upon themselves to shrug off the apathy and question me. Because of that we've found some of the limitations of my heuristics. I've had to go deeper in my explanations. I've had to express more of myself for you to make a judgement about the nature of my intent. I've had to agree with weaknesses in my over-generalised argument, concerning caution and context.

See how good that is? If you're a professional tester you do this sort of thing as a job with software already! Now we can apply some of the same rules with each other.

Carefully.

PS. Many thanks to Matt Heusser for both taking my advice, and improving it at the same time.

[1] Gause, D. and Weinberg, G. (1989). Exploring requirements. New York: Dorset House.

Apathy

How do you keep up to date with the test industry?

It's a question you've heard before at job interviews. You may even have a stock reply. You read books (you have one in mind in case they ask you what you last read) but you can't really remember the content, just the concept. You attend conferences, and while you had a great time at the after-party you forgot everything you heard because the content pandered to your emotions and to what you already believe anyway. You go to meetups and you have a simply lovely time agreeing with each other for two hours over a few drinks.

Can you defend this? Can you say that you keep yourself fresh and in-touch and up-to-date and you live in a state of constant self-improvement for these reasons?

We are a core of craftspeople struggling to improve ourselves and the world around us, we question everything, including ourselves, but it seems perfectly okay to state things like "I enjoyed that talk" without being asked to defend it. When did we get into a state where we refuse to question the value of a talk or book or article? It's very nice for us to have an opinion, but talks and books and articles absorb our time and attention and they should have some value, particularly and especially if we're going to recommend them to someone else.

So next time you hear someone tell you that they enjoyed a talk ask them why it was enjoyable and how it improved them or their testing. If we keep each other honest maybe we'll create a culture of self-questioning and improvement. If you're just interested in getting out of the office and enjoying the after-parties then could I recommend a holiday instead?

Moreover if you give out advice in public, especially if you're at a paid conference, be ready to defend it. If you're reading advice, especially if you've paid to hear it, why not question what you're hearing? Preferably publicly, so everyone can benefit from your question. If someone came for an interview and told you that they're awesome would you hire them at once, or probe for evidence? You need to ask questions. Even if you don't believe in your question you owe it to yourself, to them, and to the testing industry to pose that question - and if you're being questioned you need to understand that the questioning is for your benefit, as well as everyone else's.

We've all had a quick rant at certification or factory testing or misunderstanding of testing in our careers, so let's keep ourselves to a higher standard. Let's practice what we keep telling everyone we do and question things, including anyone who claims to speak with authority on subjects that matter. Let's not sit in the Church of Testing while we hear the good word from the pulpit, let's sit in the lecture hall after the presentation of data at a science conference and strongly question and debate what we're told in search of something better. Then let's question the questioners. And any reasonable target* of questioning should not hide behind outrage or social norms, but understand that we're all just searching for what's best.

Apathy is the dark shadows where confidence tricksters hide their lies. It's the cover of night that lets nonsense dance and play in the piazza of our industry. I now wonder if we can ever turn the lights on.


*Reasonable targets include anyone who speaks or writes with apparent authority or even strong opinion on a subject, including in response to anyone who speaks or writes with apparent authority or even strong opinion on a subject. The way that person should be questioned will depend on who they are. An industry leader should probably be more resilient than a testing newbie. Special attention should be given to anyone selling something. I'm giving the advice "be human, be an adult, have tact and try to communicate"; mainly to stem the flow of questions and comments on how we should all be nice to each other in case people become frightened to be involved.

Testers, Developers and Coders

What's the difference between a tester and a developer?

I originally looked at testing and development and the people that do it, like this:

I then migrated to believe in an overlap. Testers can sometimes develop, and developers can sometimes test.

Then came an important realisation. That testers are developers, insofar as they are part of the development of the software. Hopefully a well-used and important part.

However, there's a secret truth about this diagram. The strange thing about testing is that it doesn't require any particular skill, it just requires some sense of purpose. The desire to find things out about the product. When a coder compiles their code and sees a compiler error they've arguably just performed some testing. They have learned something about the product and then they can use the information to act on it. When they check their own work locally to see if it seems to behave sensibly before they put it in a build they have done testing. Informal testing, exploring the product for problems. This necessarily happens, it can't be avoided, even if they try! So where does that leave us, if coders are testers? Well, in some places they've decided that if testing can be done by anyone (which is pretty much true), then why not fire all of the testers and re-name the "developers"? Test is dead! Except that it's not, it's just being done by someone else.

Consider coders doing testing like bleach. It's potentially dangerous, but can be used for good. Now consider a view of testing where "testing can be automated", and "acceptance criteria" are a suitable substitute for actual requirements and non-shallow testing, as an acid that, if brought close to a sensitive surface like software development, can eat away at it. Now if we mix the bleach of tester-coders with the acid of testing ignorance we get a cloud of chlorine gas, choking the life out of coders, testers, software users, and anyone else who comes close enough. It's not likely to kill anyone in small enough doses, which is why it's possible to carry on with fake or bad testing especially in a big company where there's enough room for the gas to dissipate a little.

That's where the test expert lives. We are wardens of the health and safety of the people who design, build and use software. We ensure that the bleach is used responsibly in a well-ventilated area, and we keep dangerous acids locked away in a cupboard. We do this by not just exclusively testing, or concentrating on testing, but being skilled and tenacious in our pursuit of the truth and making it palatable enough to be realised by those blinded by their fantasies about their software. Don't strive to have testing achieved, strive to achieve good testing.

Improving Your Test Language - Automation

We think in language, and we communicate using language. Well, we perform a set of string translations that give the affordance of some meaning to someone else. I often look to Michael Bolton for clarity in the way we speak and write about testing, and his recent TestBash video is directly on the subject. I thought, as I hadn't updated my blog in months, that I'd post a little about the disjointed way some people in the industry talk and write about testing and how you might avoid doing the same - or if you want to continue doing the same understand the pitfalls of the language you're using and what you might be communicating to other people. A fair warning: this won't be news to many of you.

Automation

There is no automated testing. Automated / Manual Testing is a long-standing and extremely adhesive set of terms, but they should still be treated with due care. Testing is a human performance and cannot (currently) be automated. Manual Testing really just means "testing with your hands". In terms of software testing it cannot mean "testing without tools", in the sense that tools are required to interact with the product. One might describe hands and eyes as tools, or the computer itself, the screen, the hardware, the operating system, peripherals, the browser, and so on. These are tools that help you (enable you) to test the software.

I think that to most people automation means having the computer do a check and report the findings (automatic checking). At some point someone needs to design the checks and the reports. Also at some point someone has to read the results, interpret them, and assign them meaning and significance. These are points of access for human interaction with the tools being used in an example of tool-assisted testing.

That point is important - tool-assisted testing. All software testing is tool-assisted to some degree - and when one realises this it no longer splits testing neatly into two boxes where one is manual testing (executing test cases and some of "The Exploratory Testing") and the other is automated testing (a suite of automatic checks). Unfortunately it's a bit more complex and complicated than that. We need to look at testing as humans trying to find information. From that starting point we introduce tools because their benefits outweigh their costs. Well-written and maintainable automatic check tools can be really useful, but we must remember their costs - not just in terms of up-front financial costs, time costs, opportunity costs, maintenance costs, training costs and so on, but because a tool is an abstraction layer between the user and the software that introduces abstraction leaks. Automation (automatic checking) does not "notice" things it hasn't been strictly programmed to notice. Nor does it use the same interface to the software as a human does (although hopefully it's similar in ways that matter). Nor can it find a problem and deep-dive - meaning that it cannot interpret a false positive or false negative result. Nor can it assess risk and make meaningful decisions. Nor much of anything a tester does. Nor can it do much that a tester cannot do with "manual tools" (tools that aren't automatic check execution systems) given enough time. An automatic check suite "passing" just means that the coded checks didn't report a failure, not that suitable coverage was achieved.

This should empower you. It breaks a dichotomy of monolithic terms into a rainbow of possibilities. This should give you free reign to consider tools in your testing to make your testing more powerful, more fun, more efficient, even make testing possible... at a cost that you've thought about. Some of these costs can be reduced in the same way that testing itself can be improved - modularity and simpleness. Choosing simple, small or established tools like PerlClip, LICEcap, logs and log alerting systems, and one of my favourites: Excel can give you flexibility and power. You can pick the tools that suit your context - your favourite tools for your testing will differ greatly to mine, and for each time you test. I might use Fiddler today, Wireshark tomorrow, and Excel the next.

You don't have to choose between "Automated Testing" and "Manual Testing". They don't really exist. Put yourself in charge of your own testing and leverage the tools that get you what you want. Remember, though, that the tools you've chosen to help your processes will affect the processes you choose.

The Paucity of Pass/Fail

Pass and Fail seem to be ubiquitous terms in the testing industry. "Did the test pass?" and "How many failures?" seem to be innocent questions. But what, exactly, does pass or fail mean?

What does "pass" mean to you?

Think about that question for a second, before you move on.

Pass/Fail Reports

Checks

Let's say we're talking about a check done by a tool (sometimes called "automation"). We look at the results of that check, and it reports "PASS". What does that mean?

Let's look at this:

That's the result of a custom computer program I wrote for this automation example.

I think most people would be forgiven for thinking that this means that there is a test that's been coded that checks the login page to with a wrong password to ensure that it doesn't let the user into the system, and it passed. Most people would be forgiven for then thinking "well, we've testing logging in with an incorrect password".

If you (yes you) are a tester, then you are not the "most people" I was referring to. You should be highly suspicious of this.

What does pass mean to you now?

Tests

Let's look at this:

Tester	Test	Result
Chris	Logging in with the wrong password	PASS

This is a report given by me, a tester, for the sake of this example. I'll tell you that a human wrote all of this. Look at that PASS. Surely anything that green couldn't be wrong? If you were most people, you could be forgiven for thinking that the "logging in with the wrong password" test has passed. The coverage has been achieved. The testing has been done.

If you (yes, still you) are a tester, then you are not most people, and you should be highly suspicious of this.

So What?

Okay, testers, let's focus on those passes.

Checks

What does pass mean to you now?

To most people "pass" often means "as far as that test is concerned, everything's okay". Sometimes it means "that's been fully tested and is okay". Often it means "we don't have to worry about that thing any more".

But let's take a closer look. That automated test suite I showed you, the one with one test in it? It says that the "Invalid Login - Wrong Password" test passed. But it's not a full test, it's a check. What is it actually doing? I mean what does the "Invalid Login - Wrong Password" test do?

Well we know what it doesn't do. It doesn't investigate problems, evaluate risk, take context into consideration, interpret implicature, or do anything except what it was told to do. Maybe we investigate further and find out from the developer that what it does is enter an invalid login by having the computer enter "test1" into the username field and "password123" (which isn't test1's password) into the password field. Let's say that if, after clicking the "Login" button, the "Invalid Password" text appears, then it reports a pass, otherwise it reports a fail.

What does pass mean to you now?

Well, the explanation of the code means that the code that checked it returned a particular value (PASS) based on a specific check of a specific fact at a specific time on a specific platform, on this occasion.

Can we still have that happy feeling of "coverage" or "not having to worry" or "the testing being done"? Well, of course not. Here are some other things that would cause the code to return a "PASS" value and invalidate the test:

• The test data is wrong, or failed to load properly, and doesn't include a test1 user at all
• The "Invalid Password" text ALWAYS appears for any password
• The "Invalid Password" text appears for every first attempted login
• The text "Invalid Password" is hidden on the screen, but the checking system finds it in the DOM and reports it as found
• The text that appears for a valid password entry has been copy-pasted and is also "Invalid Password"
• The text "Invalid Password" appears elsewhere on the page after a valid login

These are scenarios that are isomorphic to the check's observed behaviour. That is to say that what the check "sees" appears the same in all of these cases, meaning that the check doesn't actually check for a wrong password login, it only checks for specific text after a specific event on a system with an unknown platform and data.

This means that for all of these cases the check reported a pass and there's a serious problem with the functionality.

We might say think because a computer said "pass" there is no problem. However, it might be that there are problems the check is not coded to do, or there may be a problem that the check does not describe because it's badly written, or something unexpected happened.

What does pass mean to you now?

Here's the actual real Ruby code I wrote for this automation example:

1 2 3 4 5 6

puts "Running Test Suite..." puts "" puts "Test: 'Invalid Login - Wrong Password' => PASS" puts "" puts "1/1 Tests run. 1 passes, 0 failures." puts "\n\n\n\n"

What does pass mean to you now?

Tests

Okay let's move on to that tester's report. A tester did the testing this time, and it passed! But while testers are intelligent and computers are not, they are frequently less predictable. What did the tester ACTUALLY do? Well, let's ask them.

Well, let's say that they said that they first checked the test data for a test1 user, and tried a valid login to confirm this. The system displayed "valid password" and gave them access.

Let's say that they said that they logged out then tried a bad password and found that it prevented them from logging in, and gave a useful, expected error message.

Let's say that they said that they tried to repeat the test a few times, and found the same behaviour.

What does pass mean to you now?

Feel better? I kind of feel better. But I think we know enough to question this, by now. What scenarios can you think of where the testing didn't find a problem related to this simple test idea? Here's a few that represent false positives or false negatives:

• The message doesn't display (or some other problem) on certain browsers
• The message doesn't display if you click the Login button twice
• When trying to log in with different passwords the system was just presenting the same screen instead of trying to log in
• The tester is testing an old version of the software that works, but the functionality is broken in the latest version.

Of more interest here are some that represent problems that weren't found:

• Every time the tester fails to log in it increments a "failed logins" value in the database. It's stored as a byte, so when it reaches 255 it throws a database error.
• The value mentioned above is responsible for locking out the user after 10 tries, so after 10 tries it ALWAYS displays "Invalid Login" on the screen, even with a valid login.

It's a fun experiment thinking of all the ways the tester didn't find existing problems while reporting a pass.

Guess what I (the tester) actually did to test the login page? That's right, nothing. I made it up. You should never have trusted me.

And what wasn't I testing? What system was I trying to test? How important is it that this works? Is it a small website to sell shoes, or a government site that needs to withstand foreign attacks?

A Passing Interlude

So let's review our meaning of "pass".

It seems to give the impression of confidence, coverage and a lack of problems.

It should give the impression of no found problems - which by itself is of exceedingly little value. Unless you know what happened and why that's important as far as "no found problems" is concerned you can't tell the difference between "good coverage and risk assessment" and "I didn't actually do any testing". Remember my test report, and my big green PASS? I didn't do any testing. The "PASS" by itself has no value. A non-tester might try one happy-path test and write PASS on their report having done no real investigation of the system.

If you're interested in a better way to report your testing then I recommend this Michael Bolton post as a jumping off point.

"PASS" is closer to meaning "Whatever we did to whatever we did it to, with however much we understand the system and however much we understand what it's supposed to do and whatever capability we have to detect problems we did not, on this occasion, with this version of this software on this platform find any problems."

I've focused on "pass" here, to show how weak a consideration it can be, and how much complexity it can obscure, but I'm going to leave it as homework to consider your version of "fail". What does "fail" mean to you? Do you use it as a jumping off point for investigations? Why don't you use "pass" as a similar jumping off point? How are your coded checks written - hard to pass or hard to fail? Why?

What Do I Do?

Remember what you're interested in. Don't chase the result of checks, chase down potential problems. Investigate, consider risk, empathise with users and learn the product. Consider problems for humans, not pass/fail on a test report. Use your sense and skill (and ethics) to avoid the pitfalls of dangerous miscommunication.

One of our jobs as testers is to dispel illusions people have about the software. People have illusions about the software because of various levels of fear and confidence about the product. Let's not create false fear and false confidence in our reporting - understand what pass and fail really are and communicate your testing responsibly.

What does pass mean to you now?

It's Just Semantics

Why is it so important to say exactly what we mean? Checking vs testing, test cases aren't artefacts, you can't write down a test, best practice vs good practice in context - isn't it a lot of effort for nothing? Who cares? 

You should. If you care about the state of testing as an intellectual craft then you should care. If you want to do good testing then you should care. If you want to be able to call out snake oil salesmen and con artists who cheapen your craft then you should care. If you want to be taken seriously then you should care. If you want to be treated like a professional person and not a fungible data entry temp then you should care.

Brian needs to make sure that their software is okay. The company doesn't want any problems in it. So they hire someone to look at it to find all the problems. Brian does not understand the infinite phase space of the testing problem. Brian wants to know how it's all going - if the money the tester is getting is going somewhere useful. He demands a record of what the tester is doing in the form of test cases because Brian doesn't understand that test cases don't represent testing. He demands pass/fail rates to represent the quality of the software because Brian doesn't understand that pass/fail does not mean problem/no problem. The test cases pile up, and writing down the testing is becoming expensive so Brian gets someone to write automated checks to save time and money because Brian doesn't understand that automated checks aren't the same as executing a test case. So now Brian has a set of speedy, repeatable, expensive automated checks that don't represent some test cases that don't represent software testing and can only pretend to fulfil the original mission of finding important problems. I won't make you sit through what happens in 6 months when the maintenance costs cripple the project and tool vendors get involved.

When we separate "checking" and "testing", it's to enforce a demarcation to prevent dangerous confusion and to act as a heuristic trigger in the mind. When we say "checking" to differentiate it from "testing" we bring up the associations with its limitations. It protects us from pitfalls such as treating automated checks as a replacement for testing. If you agree that such pitfalls are important then you can see the value in this separation of meaning. It's a case of pragmatics - understanding the difference in the context where the difference is important. Getting into the "checking vs testing" habit in your thought and language will help you test smarter, and teach others to test smarter. Arguing over the details (e.g. "can humans really check?") is important so we understand the limitations of the heuristics we're teaching ourselves, so that not only do we understand that testing and checking are different in terms of meaning, but we know the weaknesses of the definitions we use.

So, we argue over the meaning of what we say and what we do for a few reasons. It helps us talk pragmatically about our work, it helps to dispel myths that lead to bad practices, it helps us understand what we do better, it helps us challenge bad thinking when we see it, it helps us shape practical testing into something intellectual, fun and engaging, and it helps to promote an intellectual, fun and engaging image of the craft of testing. It also keeps you from being a bored script jockey treated like a fungible asset quickly off-shored to somewhere cheaper.

So why are people resistant to engaging with these arguments?

It's Only Semantics

When we talk about semantics let's first know the difference between meaning and terminology. The words we use contain explicature and implicature, operate within context and have connotation. 

Let's use this phrase: "She has gotten into bed with many men".

First there's the problem of semantic meaning. Maybe I'm talking about bed, the place where many people traditionally go in order to sleep or have sex, and maybe you think I'm talking about an exclusive nightclub called Bed.

Then there's the problem with linguistic pragmatic meaning (effect of context on meaning). We might disagree on the implication that this women had sex with these men. Maybe it's an extract of a conversation about a woman working with a bed testing team where she's consistently the only female and the resultant effects of male-oriented bed and mattress design.

Then there's the problem with subjective pragmatic meaning. We might agree that she's had sex with men, but might disagree on what "many" is, or what it says about her character, or even agree that I'm saying that she's had sex with many men where I am implying something negative about her character but you disagree because you've known her for years and she's very moral, ethical, charitable, helpful and kind.

So where we communicate we must understand the terms we use (in some way that permits us to transmit information, so when I say "tennis ball" you picture a tennis ball and not a penguin), and we must appreciate the pragmatic context of the term (so when I say "tennis ball" you picture a tennis ball and not a tennis-themed party).

When we say "it's only a semantic problem" we're inferring that the denotation (the literal meaning of the words we use) is not as important as the connotation (the inference we share from the use of the words).

Firstly, to deal with the denotation of our terminology. How do we know that we know what we mean unless we challenge the use of words that seem to mean something completely different? Why should we teach new people to the craft the wrong words? Why do we need to create a secret language of meaning? Why permit weasel words for con artists to leverage?

Secondly, connotation. The words we use convey meaning by their context and associations. Choosing the wrong words can lead to misunderstandings. Understanding and conveying the meaning IS the important thing, but words and phrases and sentences are more than the sum of their literal meaning, and intent and understanding of domain knowledge do not always translate into effective communication. Don't forget that meaning is in the mind of the beholder - an inference based on observation processed by mental models shaped by culture and experience. It's arguments about the use of words that lead to better examination of what we all believe is the case. For example some people say that whatever same-sex marriage is it should be called something else (say, "civil union") because the word marriage conveys religious or traditional meaning that infers that it is between a man and a woman. The legal document issued is still the marriage licence. We interpret the implicature given the context of the use of these words which affects the nature of the debate we have about it. We can argue about "marriage" - the traditional religious union, and we can argue about "marriage" - the relationship embodied in national law and the rights it confers. We can also talk about "marriage" - the practical, day to day problems and benefits and trials and goings on of being married.

Things Will Never Change

Things are changing now. Jerry Weinberg, James Bach, Michael Bolton and many others' work is rife with challenging the status quo, and now we have a growing CDT community with increasing support from people who want to make the testing industry respected and useful. There's work in India to change the face of the testing industry (Pradeep Soundararajan's name comes to mind). There's STC, MoT, AST and ISST. The idea that a problem won't be resolved is a very poor excuse not to be part of moving towards a solution. Things are always changing. We are at the forefront of a new age of testing, mining the coalface of knowledge for new insight! It's an incredibly exciting time to be in a changing industry. Make sure it changes for the better.

It's The Way It's Always Been

Yes, that's the problem.

I'll Just Deal With Other Challenges

Also known as "someone else will sort that out". Fine, but what about your own understanding? If you don't know WHY there's a testing vs checking problem then you're blinding yourself to the problems behind factory testing. If you don't investigate the testing vs checking problem then you won't know why the difference between the two is important to the nature of any solution to the testing problem, and you're leaving yourself open to be hoodwinked by liars. Understand the value of what you're doing, and make sure that you're not doing something harmful. At the very least keep yourself up to date with these debates and controversies.

There's another point to be made here about moral duty in regard to such debates and controversies. If you don't challenge bad thinking then you are permitting bad thinking to permeate your industry. If we don't stand up to those who want to sell us testing solutions that don't work then they will continue to do so with complete repudiation. I think that this moral duty goes hand-in-hand with a passion for the craft of testing, just as challenging racist or sexist remarks goes hand-in-hand with an interest in the betterment of society as we each see it. It's okay to say "I'm not going to get into that", but if you claim to support the betterment of the testing industry and its reputation as an intellectual, skilled craft then we could really use your voice when someone suggests something ridiculous, and I'd love to see more, innovative approaches to challenging bad or lazy thinking.

"Bad men need nothing more to compass their ends, than that good men should look on and do nothing." - John Stuart Mill

Sounds Like A Lot Of Effort

Testing is an enormously fun, engaging, intellectual activity, but it requires a lot of effort. Life is the same way. For both of these things what you get out of it depends on what you put in.

Arguments Are Just Making Things Less Clear/Simple

This is an stance I simply do not understand. The whole point of debate is to bash out the ideas and let reason run free. Clear definitions give us a starting point for debate, debate gives us rigour to our ideas. I fail to see how good argument doesn't make things better in any field.

As for simplicity, KISS is a heuristic that fails when it comes to the progress of human knowledge. We are concerned here with the forefront of knowledge and understanding, not implementing a process. Why are we so eager to keep it so simple at the expense of it being correct? I've talked to many testers and I've rarely thought that they couldn't handle the complexity of testing, as much as they could the complexity of life. I don't know who we're protecting with the idea that we must keep things simple, especially when the cost is in knowledge, understanding and the progression of our industry.


P.S.
I've been reluctant to release this post. It represents my current thinking (I think!), but I think it's uninformed which may mean that it's not even-handed. Linguistics and semantics are fairly new to me. I'm reading "Tacit and Explicit Knowledge" at the moment, I may come back and refine my ideas. Please still argue with me if you think I'm wrong, I have a lot to learn here.

Best Practices Aren't

Best Practices Aren't

I assume that I'm directing myself at a group of people who know that there's no such thing as a Best Practice, and that we shouldn't use the term. There's a bit of consternation around this, some people defending the term whilst understanding the nature of contextual value of practices.

I'm working under the assumption that you've read this: http://www.satisfice.com/blog/archives/27

Of course there are no Best Practices! Of course practices only apply to a particular context! Okay, with that under our belt I'd like to talk about why we shouldn't use the term at all, and why it matters that we shouldn't.

It's A Trap!

Saying what we mean is important. We can call poison "food" and assume that tacitly we all know it's actually poison, really - until the amateur or black-and-white thinker wanders in and eats it, or some dishonest salesperson sells it to someone as food (after all, that's what we're calling it). In the same way we can all agree that "best" just means a particularly good thing to do in the context, but when people use "best practices" as a benchmark, ignoring the subtleties in the change in context, bad things can happen. Someone can look at a standard in one company and assume it will work for them because they're in the same industry - but there's a lot more to context than that. There's methodologies used, hours worked, company culture, relationships between people and teams, budget, development speed, even the geographical location of the office building. So why do we need this secret society of people who understand that "Best Practice" doesn't really mean "best"? Especially when we can talk about practices without using that term in the first place? Maybe it's some sort of egotistical elitism or some desire to find tranquil simplicity in a complex world, but either way it should affront your professional ethics to want to be part of the Secret Society of Avoidable Confusion.

It's Lazy Thinking

Understanding the flaws in a practice - the contexts in which it won't work or has little value - is important. Knowing what we should do is better than assuming that emulating someone else's practices will be just fine. It's thinking like this that permits tool vendors to sell snake oil cure-alls for the testing problem, and permits the persistence of worthless certifications. We can say "of course it's for a particular context" but that's because we don't want to have to think about the effect of context on the value of our practices because then we'd have to do some work. If you're the sort of person that believes that "automation" is more than an automatic check execution system to replace good, exploratory testing then you understand the importance that context has on the value of a practice. So don't let your brain go on holiday just because someone stuck the word "best" in front of "practice" and claimed it as some sort of benchmark of greatness, even if they permit you the luxury of implicit contextual value. Think critically, and don't let anyone stop you from doing so.

It Replaces Discussion

The "of course it's for a particular context" argument is the end of the discussion of our practices when we hold Best Practices up as a benchmark. Thinking about our practices, and their value, with the influence of practices we know work well elsewhere is not something to avoid or be ashamed of. There seems to be some resistance to the idea of eliminating best practices because of laziness, and some fear of "reinventing the wheel" - except that we're not re-inventing a wheel, we're adapting the wheel, changing the wheel, assigning cost to the wheel, using a different kind of wheel, examining how suitable the wheel is for our purpose and perhaps deciding that we don't need a wheel. "Don't re-invent the wheel" does not mean that we should use stone cylinders with a hole in them on our motor vehicles. Use the right wheel for what you need to do, even if someone says that a particular wheel is the "Best". Use your awesome tester powers of critical thinking.

It's Not What "Best" Means

"Best" is a superlative. The form of the adjective "good" that infers that it is the greatest of any other possible degree of something. When you say (without hyperbole) that "this is the best-tasting sandwich I've ever eaten" you are saying that there is no sandwich that you have ever eaten that tastes better than this one. Best Practice simply does not make sense, even with context factored in. Even for a particular context there is no Best Practice that we can know is the best one. "Best Practice for this context" means "There is no practice that exists or could be thought of that could be better than this one for this context" - which is quite the claim! Don't let people get away with that sort of thing.

An Alternative

If you really need a short-handed way to say "Best Practice" but actually mean something more like "good practice for a particular context" here are some ideas:

• (I've found it to be an) Effective Practice
• Cool Idea
• The "<your name here>" Method
• (My) Favourite Practice (for)

Or alternatively you could just talk openly and honestly about practices, where they seem to work for you, and where and how they could be applied to good affect.

STP Podcast - Automation In Testing with Richard Bradshaw

Listen here:
https://www.youtube.com/watch?v=UonH_5-X14k

This was a good listen! So good I'm inspired to comment in more space than can fit in a tweet. I do agree with basically everything mentioned, which is a shame, but it did bring up ideas for me that I hadn't considered and ways of expressing things I hadn't heard yet, which is a treat.

Forgive the quality, I rushed the whole post.

Automation Is A Tool

Richard is clear and passionate on this topic, and I could not agree with him more. Automation is a tool, and as a tool it has to serve a test strategy in some way. Automation is so often seen as chasing some benefit that doesn't exist. It's like buying a new battery-powered electronic drill purely under the assumption that it will save you time with your DIY. Well it depends on what you're going to use it for - why do you need a drill, and why does it need to be this drill? A drill doesn't mean you don't have to drill holes any more. It also requires maintenance. It requires skill to use. It can enhance your ability to drill holes, and it can enhance your ability to make mistakes and do damage. It has disadvantages over other hole-boring tools for certain tasks. It requires exploratory learning during its use, examining and interpreting the results.

Exploratory Automation

I agree with the statements in the podcast. I'd go as far as to say that there's no such thing as an exploratory tool. I'd accept "pseudo-exploratory", maybe. Exploration requires learning, and only humans can learn in that way. For a tool we have to encode how it interacts with the product, what oracles it can use and how to use them, how to make decisions... and we cannot encode it to cope with the implicature of that information nor can we code it to be inspired to do anything we haven't yet thought of. Humans can notice a symptom, follow it, design new tests, learn from the results, design further tests, and so on leveraging heuristic principles to find problems or build better mental models. They can get inspired by results and use that to explore new ideas based on their understanding of risk (heuristics such as "it's bad when people get hurt" that are hard to teach to a machine).

Metrics

"How do I measure my manual testing and have visibility into the value of my manual testers?"
"How do I have visibility into the value of my automation?"

Richard gave a great answer to this.

There is no way to usefully quantitatively measure the value (someone's regard of the importance or usefulness of something) of a tool. That's the problem, it's subjective. Don't automate to save money. Automate when it serves the test strategy. Are you willing to invest in the quality and speed of the information  testing provides, or are you looking to replace an expensive but necessary service? You wouldn't buy a new IDE for your development team with fancy code tools so that you could fire some of your developers and get return on investment, so you shouldn't do it with testing (unless your testers are THAT replaceable, in which case your problems can't be solved with a tool). As Richard says, "Automation is there to serve the team".

I'm always curious as to why people want to value their tools like that. I understand the hesitation if someone is buying an automated check execution tool under impossible promises from a vendor, but I'm more interested in reality. Let me ask this: How do you get visibility into the value of your car? Your car (or a car you might buy if you don't have one) is a tool that you can use. Well, you can look at the cost of the car, maintenance and fuel but you have to measure it against how useful it is. How do you measure how useful something is? It's a subjective matter. What do you use your car for? Could you do without a car? What's the difference in utility between having the car and not having the car? What are the drawbacks? How do you factor those into the utility? What about things you haven't thought of?

Other

I also really liked Richard's point around here of "automation is dumb" and that it doesn't need to be intelligent because we have intelligent testers to use it.

This wasn't even half of the content of the podcast, so I'd recommend listening to the whole thing. They go on to discuss automation tools beyond automatic check execution and how to use them to inform mental models, the kinds of people who do different types of testing, reporting and so much more. I'd like to write about this too but I only have so much time!

Thanks to STP, Mark Tomlinson and Richard Bradshaw for a great podcast!

I Failed The ISTQB Practice Exam - Part VI & Final Thoughts

Series: Part I, Part II, Part III & IV, Part V

I was on 19/36...

Part VI - Tool Support For Testing

37. From the list below, select the recommended principles for introducing a chosen test tool in an organization? 

1. Roll the tool out to the entire organization at the same time. 
2. Start with a pilot project. 
3. Adapt and improve processes to fit the use of the tool. 
4. Provide training and coaching for new users. 
5. Let each team decide their own standard ways of using the tool. 
6. Monitor that costs do not exceed initial acquisition cost. 
7. Gather lessons learned from all teams. 

a) 1, 2, 3, 5.
b) 1, 4, 6, 7.
c) 2, 3, 4, 7.
d) 3, 4, 5, 6.

My Answer: B - only one without 3, except 6!
ISTQB Says: C

Obviously 3 is stupid, so (b) must be the answer! Except that it also has 6, which is rediculous. 1 would vary depending on the tool (does the CEO need RapidReporter?), 2 might be important if it's a big tool, 3 means being shaped by our tools which is dangerous, 4 might be necessary if they haven't used the tool before and it's complex or we want to use it in a certain way, 5 is a possibility (I don't teach anyone how to use Excel or a log viewer in a specific way, for example), 6 seems weird and 7 seems like a good idea. I suppose if it wasn't for 3 I might have picked C.

And who's the guy recommending these principles? Come on, someone must have had to. Unless they're sent down from on high etched into stone tablets or something.

Score: 19/37


38. Which one of the following best describes a characteristic of a keyword-driven test execution tool?

a) A table with test input data, action words, and expected results, controls execution of the system under test
b) Actions of testers recorded in a script that is rerun several times.
c) Actions of testers recorded in a script that is run with several sets of test input data
d) The ability to log test results and compare them against the expected results, stored in a text file.

My Answer: A
ISTQB Says: A

Another point for yours truly as he deftly answers the challenge of keyword-driven test execution by choosing the only answer that contains any test execution.

Score: 20/38


39. Which of the following is NOT a goal of a Pilot Project for tool evaluation?

a) To evaluate how the tool fits with existing processes and practices.

b) To determine use, management, storage, and maintenance of the tool and test assets.

c) To assess whether the benefits will be achieved at reasonable cost.

d) To reduce the defect rate in the Pilot Project.

My Answer: D
ISTQB Says: D

I wonder what this all has to do with testing? I can name animals but I'm not a vet.

Score: 21/39

40. Below is a list of test efficiency improvement goals a software development and test organization would like to achieve. 

 

Which of these goals would best be supported by a test management tool? 

 

a) To build traceability between requirements, tests, and bugs.

b) To optimize the ability of tests to identify failures.

c) To resolve defects faster.

d) To automate selection of test cases for execution.

 

My Answer: D
ISTQB Says: A

What's a test management tool? A tool to manage testing? I do that myself. Unless you mean to manage the test project? I do that too. What does this even mean? I chose D because A is pointless, B is impossible, C is just plain false (the opposite may be true) and there are tools that can generate test cases for you based on inputs (like Hexawise).

Score: 21/40

The Result

I scored 21/40, which is 52.5%. I required 65% to pass, which would mean I'd have to have answered 26 of the 40 questions correctly.

This means that I needed to get 5 more points to have passed the exam and gotten Foundation level certification. It means that I am permitted to answer 14 of these questions incorrectly and get Foundation level certification.

I don't want to disturb anyone, but it seems to me that it's remarkably easy to get this certification provided you play ball and answer based on what you expect of them.

Final Thoughts

Well, we've all had some fun over the last few days, but there are some serious points to be made here. I got 52.5% on this exam by forgetting to answer questions, being facetious, answering honestly and not studying the course materials or syllabus. We can probably say that if I'd studied I would have passed, and so would nearly anyone else who wanted to. We can also probably say that if someone within the test industry can get such a low score it's questionable if the content of the exam.tests for useful knowledge.

I know I've berated some of the answers in this exam. I expected to. I took this, thinking it might help to clear up for me why the ISTQB is so hated by some with a little first-hand knowledge. I expected a highly academic tightly-written exam that glossed over social sciences and self-learning in exchange for repetition of terms and testing the knowledge of techniques and documentation. I found that, but what I found exceeded my own expectations of such a divide. I have enough problems with the questions in this exam to ensure no interest in the materials. Taking this test has taught me so much about the ISTQB exam already and frankly the exam paper reads like a scam. It's like renaming fruit with names of vegetables, then showing you pictures of the fruit and getting you to utter the name of the associated vegetable. Then charging you money and giving you paper in order to pass a contrived barrier to the field of testing, and everything that that implies.

Firstly, there's the quality of the questions themselves. There's no precision of language, misuse of commas, grammatical errors, terms given in confusing English, and questions that can be logically reduced to 50% guesses or solved without testing knowledge. Secondly, the content of the questions. It's well known that the ISTQB exam does not test for skill or ability. It tests for knowledge. But what knowledge, and how useful is it? Is it useful to know what equivalence partitioning is? Yes. But what about how it works, when it's used, how it's useful, when it's not useful, when it fails, in what situations it's probably more costly than valuable. What about practising on real world examples? What about defending an opinion on it, not just presenting one or being given one by the exam board? What about development of skill? None of this seems to be tested in the exam. Don't get me wrong, it may or may not be taught on the course (I've never studied for it and I've never taken it) but the exam doesn't test for it. So no matter how good the course is people who misunderstand testing will pass because of the quality of the exam questions and people who do understand testing will fail because of the quality of the exam questions.

What I found was confused, badly written, badly designed questions asking pointless things that do not test even the knowledge of a factory school course let alone an understanding of testing and how to do it. No matter what your camp, school, field, knowledge, skill or experience I'd seriously consider examining the claims of this certification versus what it actually is capable of doing. If you want to be a good tester you have to ask questions of the thing under test. Be a good tester, and ask questions of certifications. You'll come up with your own opinion, you certainly don't need me to tell you that, but you can't have an informed opinion without first informing yourself. And if you still want to take the certification, you're going to need to take the material with enough salt to risk sodium poisoning.

Let's just say this: If I show you that someone that got 100% on this exam would it positively influence your decision to hire them?

So what are your thoughts? Any insights on the questions? Anything you disagree with? Let me know in the comments, or contact me on Twitter @kinofrost.

Again, the exam and marking sheet are all under copyright owned by the ISTQB.

Good follow-ups to this post by better writers are below:

James Bach - http://www.satisfice.com/blog/index.php?s=istqb
Alan Richardson - http://blog.eviltester.com/2008/05/a-short-history-of-my-iseb-software-testing-certification-involvement.html
Rob Lambert - http://thesocialtester.co.uk/certifications-are-creating-lazy-hiring-managers/
Pradeep Soundararajan - http://testertested.blogspot.co.uk/2013/04/the-istqb-scam-and-why-you-should-sign.html

I Failed The ISTQB Practice Exam - Part V

Series: Part I, Part II, Part III & IV, Part V, Part VI and Final Thoughts

I was on 15/28...

Part V - Test Management

29. Which of the following best describes the task partition between test manager and tester?

a) The test manager plans testing activities and chooses the standards to be followed, while the tester chooses the tools and controls to be used. 
b) The test manager plans, organizes and controls the testing activities, while the tester specifies, automates and executes tests. 
c) The test manager plans, monitors and controls the testing activities, while the tester designs tests. 
d) The test manager plans and organizes the testing and specifies the test cases, while the tester prioritizes and executes the tests. 

My Answer: None / it depends
ISTQB Says: B

It depends on the company. In A sometimes testers choose tools to be used, in D sometimes the tester specifies test cases and the manager prioritizes testing.
What's really interesting go me is the answer in B (also in C) that the manager controls the testing activities. Only if you hire dead testers and tie strings to them like a macabre puppeteer. Hey testers, do you have any control over your test activities? That is to say when you're doing the things you do to test do you have any control over the things you do to test? Does your manager have any direct control over this at all?

Well you're wrong because it's B.

Score: 15/29



30. Which of the following can be categorized as product risks?

a) Low quality of requirements, design, code and tests.
b) Political problems and delays in especially complex areas in the product.
c) Error-prone areas, potential harm to the user, poor product characteristics.
d) Problems in defining the right requirements, potential failure areas in the software or system.

My Answer: C
ISTQB Says: C

A is project, B is project, D is project, C is product. Me good test now.

Score: 16/30


31. Which of the following are typical test exit criteria?
a) Thoroughness measures, reliability measures, test cost, schedule, state of defect correction and residual risks.
b) Thoroughness measures, reliability measures, degree of tester independence and product completeness. 
c) Thoroughness measures, reliability measures, test cost, time to market and product completeness, availability of testable code.
d) Time to market, residual defects, tester qualification, degree of tester independence, thoroughness measures and test cost.

My Answer: A
ISTQB Says: A

What are "test exit criteria"?  For me that's things like "end of the session" or "I'm bored now" or "Not much more I can do here" or "time for a break". Things like that. How do we measure thoroughness? Seriously, measure my thoroughness, I dare you. Then measure my reliability. I like to use a tricorder. State of defect correction is a test exit criterion?
This doesn't make any sense. I'm so confused. Oh, the answer was a guess.

Score: 17/31

32. As a Test Manager you have the following requirements to be tested: 
Requirements to test: 
R1 - Process Anomalies – High Complexity 
R2 - Remote Services – Medium Complexity 
R3 – Synchronization – Medium Complexity 
R4 – Confirmation – Medium Complexity 
R5 - Process closures – Low Complexity 
R6 – Issues – Low Complexity 
R7 - Financial Data – Low Complexity 
R8 - Diagram Data – Low Complexity 
R9 - Changes on user profile – Medium Complexity 


Requirements logical dependencies (A -> B means that B is dependent on A): 

 

How would you structure the test execution schedule according to the requirement dependencies? 

a) R4 > R5 > R1 > R2 > R3 > R7 > R8 > R6 > R9.
b) R1 > R2 > R3 > R4 > R5 > R7 > R8 > R6 > R9.
c) R1 > R2 > R4 > R5 > R3 > R7 > R8 > R6 > R9.
d) R1 > R2 > R3 > R7 > R8 > R4 > R5 > R6 > R9.

My Answer: I wouldn't.
ISTQB Says: C

Who in the name of buttery buttocks structures a test execution schedule ahead of time? If you've worked on a serious product and had a test execution schedule (a schedule for executing tests) that actually says when what tests are executed and stuck to it I'll give you 20 quid. Why are we assuming that this document is accurate? Who wrote it and why? Are these really the dependencies? According to whom? What about changes to the project? What about parallel activities or early availability of functionality or documentation? Can we express the logical relationship between explicit requirements with an arrow? Can we do it with implicit/tacit requirements at all? Hint: no. There's an infinite number of places where this schedule can fail. Is it important to schedule by dependency in whatever this example would be in real life? These are the interesting questions.
So no, I would not just write a test execution schedule, I'd sketch a plan based on things like availability, testability, deadlines, deliverables, and not just complexity but RISK.
I'd like to do an experiment. Change the question to just show the diagram, the answers, and a big question mark and see if the pass rate for this question remains the same.

Score: 17/32

33. What is the benefit of independent testing?

a) More work gets done because testers do not disturb the developers all the time.
b) Independent testers tend to be unbiased and find different defects than the developers.
c) Independent testers do not need extra education and training.
d) Independent testers reduce the bottleneck in the incident management process.

My Answer: D?
ISTQB Says: B

What is independent testing? Independent of what? Freedom of thought? I could recommend some books and scientific papers indicating that independent testers have a raft of biases. And what evidence is there that they find "different" defects? Different how? Do they necessarily and all the time find only different defects?This question is so infuriating.

Score: 17/33

34. Which of the following would be categorized as project risks?

a) Skill and staff shortages.
b) Poor software characteristics.
c) Failure-prone software delivered.
d) Possible reliability defect (bug).

My Answer: All of them, obviously
ISTQB Says: A

Okay, if you have a skill/staff shortage that could run a risk to your project, depending on what the skill and staff shortages are. I mean, we have a huge skill shortage when it comes to arc welding, architecture and railway maintenance but it's never seemed to come up.
If the software has "poor characteristics" then that could be a project risk. For example if the product has the poor characteristic of "not very testable" then testers may not be able to provide fast useful feedback to test clients which means that people are making uninformed decisions which is a project risk.
Delivering failure-prone software is a project risk. It's a risk to the success of the project. Delivering a failure-prone product is definitely a risk to the success of the project. I don't know how else to put this.
A "possible reliability defect" is a risk (as it's only a possible bug) and it threatens the success of the project so it's a project risk.

Am I just being dense? Anyway, the correct answer is A.

Score: 17/34

35. As a test manager you are asked for a test summary report. Concerning test activities and according to IEEE 829 Standard, what should you consider in your report? 

a) The number of test cases using Black Box techniques.
b) A summary of the major testing activities, events and its status in respect of meeting goals.
c) Overall evaluation of each development work item.
d) Training taken by members of the test team to support the test effort.

My Answer: B
ISTQB Says: B

I don't know the IEEE 829 standard. That didn't help as I wasn't allowed to look it up. I've never needed to. I've never worked in an IEEE 829 standard environment. And if I did the company would ensure I knew IEEE 829 well enough. So why is there a question on it here? Do they teach IEEE 829 for the exam? But hey, if there's anything that says "efficiency" is standardisation of test documentation, right? How about "report what's needed to the right person in the right way at the right time"? I like that as a standard. I'll call it Common Sense 829.
I guessed the answer they wanted. The actual answer is: you'd consider them all if you felt it was necessary to summarise your testing. Say you needed to train your test team to use a helpful tool in order to better test the product? I'd include that in my report to a manager who wants to know what I've been doing with my time and why more testing wasn't done.

Score: 18/35

36. You are a tester in a safety-critical software development project. During execution of a test, you find out that one of your expected results was not achieved. You write an incident report about it. What do you consider to be the most important information to include according to the IEEE Std. 829? 

a) Impact, incident description, date and time, your name.
b) Unique id for the report, special requirements needed.
c) Transmitted items, your name and you’re feeling about the defect source.
d) Incident description, environment, expected results.

My Answer: A?
ISTQB Says: A

Again, I don't know the standard. A is the only one that contained both description and impact. D was another candidate as it contained environment and expectation (oracle information). What  the gibbering testicle are "transmitted items" anyway? I don't know "you're" feeling on the matter.*

* Remember, this is the practice exam directly downloaded from the ISTQB's own website, provided by them as a workable example.

Score: 19/36


Next time, the thrilling conclusion and my final thoughts! See you then!

I Failed The ISTQB Practice Exam - Part III and IV

Series: Part I, Part II, Part III & IV, Part V, Part VI and Final Thoughts

Welcome back! Static Techniques is short, so I've thrown in Test Design Techniques at no extra charge. I was on 7/13...

Part III - Static Techniques

14. Which of the following are the main phases of a formal review?

a) Initiation, status, preparation, review meeting, rework, follow up.
b) Planning, preparation, review meeting, rework, closure, follow up.
c) Planning, kick off, individual preparation, review meeting, rework, follow up.
d) Preparation, review meeting, rework, closure, follow up, root cause analysis.

My answer: ?

ISTQB says: C

A formal review of what? Any review of anything that conforms to a particular structure. Okay. So any of these could be the main phases of a review that is formal. The question is actually asking "which of the following are the main phases of the kind of formal review that I'm thinking of right now". I couldn't have answered this without reading the material and memorising a specific kind of formal review.

Score: 7/14

15. Which TWO of the review types below are the BEST fitted (most adequate) options to choose for reviewing safety critical components in a software project? 
Select 2 options. 

a) Informal review.
b) Management review.
c) Inspection.
d) Walkthrough.
e) Technical Review.

My answer: C & E, it depends

ISTQB says: C and E

I gave myself the marks, because I had the same answer. I don't know why these are always the most adequate options... unless they're just the most adequate out of the selection. An informal review (by my terminology as any review that is informal) can have powerful results in terms of learning. A review by management may be helpful in some circumstances - even a review of management might help. Inspection is the stupidest answer I've ever heard for anything - of course you're going to inspect the safety critical components. 1/40th of this exam rides on the answer to the question "do you think you should look at safety critical components when reviewing them?" being "yes". Walkthroughs are incredibly useful for understanding the structure, interface and purpose of components. Technical reviews are... also useful. I just picked two because the question told me to pick two, but I could debate the case for each. Oh, and what the hell is a "component" in the context of the project, and what is the project for? To replicate the feeling I'm having here imagine having George R R Martin on your radio show about improving creating writing and asking him "so, which is better, pencil or pen?".

Score: 8/15

16. Which of the following statements about static analysis is FALSE?

a) Static analysis can be used as a preventive measure with appropriate process in place. 
b) Static analysis can find defects that are not easily found by dynamic testing.
c) Static analysis can result in cost savings by finding defects early.
d) Static analysis is a good way to force failures into the software.

My answer: D

ISTQB says: D

What an utter waste of words. You don't need me to explain this to you, do you? Which of the following is FALSE about Test Technique Foo? a) It can be used to do good things, b) It can help with stuff, c) It's totally good, d) It will make your software worse by magic. Thank you, now I completely understand your knowledge of Test Technique Foo.

Score: 9/16

Part IV - Test Design Techniques

17. One of the test goals for the project is to have 100% decision coverage. The 
following three tests have been executed for the control flow graph shown 
below.

Test A covers path: A, B, D, E, G. 
Test B covers path: A, B, D, E, F, G. 
Test C covers path: A, C, F, C, F, C, F, G. 

Which of the following statements related to the decision coverage goal is 
correct? 

a) Decision D has not been tested completely.  
b) 100% decision coverage has been achieved.  
c) Decision E has not been tested completely.  
d) Decision F has not been tested completely.  

My answer: I want nothing to do with this

ISTQB says: A

Anyone that can follow a set of arrows could have answered this question, but I stuck to my principles with this one. Obviously the path A, B, D, F isn't covered by the test cases. But why should we care?
Firstly, this is a diagram and not a program. How helpful is it to have this diagram without knowing what it's modelling? What is it leaving out? What ARE these decisions? And what can we honestly and reliably say about them being "tested completely". Okay, so you've gone through your if statement and it's gone to each direction and it hasn't errored. What value does that give you? What have you learned about the system and if it's fit for use?
Secondly what on earth are these tests, and what does it mean to "cover" a path? Are the paths really covered by those tests? That's an important question. What about hidden paths? Say you're testing a web application and the session times out. Say you pull the plug out of your desktop machine. That's a path too. You can look at the decisions independently (see what conditional statements the workflow relies upon) but what does it mean to weave them all together in this graph?
I will not answer this question without my curiosity assuaged, sir. And for my impertinence I am punished by one point.

Score: 9/17

18. A defect was found during testing. When the network got disconnected while receiving data from a server, the system crashed. The defect was fixed by correcting the code that checked the network availability during data transfer. The existing test cases covered 100% of all statements of the corresponding module. To verify the fix and ensure more extensive coverage, some new tests were designed and added to the test suite. 

What types of testing are mentioned above?

A. Functional testing.
B. Structural testing. 
C. Re-testing. 
D. Performance testing. 

a) A, B and D.  
b) A and C.  
c) A, B and C.  
d) A, C and D.  

My answer: Type C, so none

ISTQB says: C

Was functional testing mentioned? Well the statement "A defect was found during testing" could be any testing. "When the network got disconnected while receiving data from a server, the system crashed" could be as part of testing I suppose, and if it is then the type of testing used wasn't mentioned.  Re-testing is mentioned, as something was tested again. Performance testing - we'll never know if someone was specifically testing for performance issues.

Score: 9/18

19. Which of the following statements about the given state table is TRUE? 

a) The state table can be used to derive both valid and invalid transitions.
b) The state table represents all possible single transitions.
c) The state table represents only some of all possible single transitions.
d) The state table represents sequential pairs of transitions.

My answer: C

ISTQB says: B

Okay. A is actually true. It could be used to derive valid and invalid transitions. It does not represent all possible single transitions that exist because this is an abstract model. Therefore C is correct. D is...

I'm wasting my time. Why is this important? How am I a better tester for this? The more I analyse what I wrote the more I question why I bothered to write it.

Score: 9/19

20. Which of the following statements are true for the equivalence partitioning test technique? 

A. Divides possible inputs into classes that have the same behaviour.
B. Uses both valid and invalid partitions.
C. Makes use only of valid partitions.
D. Must include at least two values from every equivalence partition.
E. Can be used only for testing equivalence partitions inputs from a Graphical User Interface.

a) A, B and E are true; C and D are false.
b) A, C and D are true; B and E are false.
c) A and E are true; B, C and D are false.
d) A and B are true; C, D and E are false.

My answer: D

ISTQB says: D

This sort of examines if I understand what equivalence partitioning is. Not what it's for, how it's used, when it fails, just what it is. A is basically the case, B is also basically the case. If B is true then C can't be true. D is just an arbitrary rule. E is the realm of the unhinged.

Knowing that E is probably wrong to the layman we can narrow down our choices to (b) and (d). Both contain A as true. B and C are mutually exclusive, so we don't have to know D. So with a little applied logic we can safely say that this question can be replaced with:

Does the equivalence partitioning test technique make use of both valid and invalid partitions? YES/NO.

Just trying to save you the ink, guys.

Score: 10/20

21. Which TWO of the following solutions below lists techniques that can all be categorized as Black Box design techniques?
Select 2 options. 

a) Equivalence Partitioning, decision tables, state transition, and boundary value.
b) Equivalence Partitioning, decision tables, use case.
c) Equivalence Partitioning, decision tables, checklist based, statement coverage, use case.
d) Equivalence Partitioning, cause-effect graph, checklist based, decision coverage, use case.
e) Equivalence Partitioning, cause-effect graph, checklist based, decision coverage and boundary value.

My answer: A and B

ISTQB says: A and B

Okay, cross out equivalence partitioning as it's in all of them. Decision tables split A, B and C from D and E, so the answer is either (d) and (e) or some part of (a), (b) or (c). See, we're learning testing! Now then, "checklist based" is found in (c), (d) and (e) which means that (c) cannot be correct. So it's either (a) and (b) or (d) and (e) - down to a 50% chance of guessing!

(a) and (b) contain: EP, DT, ST, BV, UC.
(d) and (e) contain: EP, CEG, CB, DC, UC, BV.

Removing common elements:
(a) and (b) contain: DT, ST
(d) and (e) contain: CEG, CB, DC

So is the answer "Decision tables and state transition" or is it "cause-effect graph, checklist based and decision coverage".

Well, the shorter list is more likely, I went for that one.

That is the entirety of the knowledge I used in answering this question. This is not an interesting or useful question.

Score: 11/21

22. An employee’s bonus is to be calculated. It cannot become negative, but it can be calculated to zero. The bonus is based on the duration of the employment. An employee can be employed for less than or equal to 2 years, more than 2 years but less than 5 years, 5 to 10 years, or longer than 10 years. Depending on this period of employment, an employee will get either no bonus or a bonus of 10%, 25% or 35%. How many equivalence partitions are needed to test the calculation of the bonus?
a) 3.
b) 5.
c) 2.
d) 4.

My answer: D

ISTQB says: D

It CANNOT BECOME NEGATIVE. The laws of nature simply preclude it from being so. God himself has written the validation - etched it into the soul of the universe. If you can count a list of items you can guess the probable "answer" to this question. Of course in reality context needed, it depends, asking questions, blah, blah.

Score: 12/22

23. Which of the following statements about the benefits of deriving test cases from use cases are most likely to be true? 
A. Deriving test cases from use cases is helpful for system and acceptance testing. 
B. Deriving test cases from use cases is helpful only for automated testing. 
C. Deriving test cases from use cases is helpful for component testing. 
D. Deriving test cases from use cases is helpful for testing the interaction between different components of the system. 

a) A and D are true; B and C are false.  
b) A is true; B, C and D are false.  
c) A and B are true; C and D are false.  
d) C is true; A, B and D are false.  

My answer: All of them, so none of the answers.

ISTQB says: A

There are situations in which all of them are most true. A can be true, B can be true, C can be true, D can be true. Depends on the test cases and the use cases and the situation. Usually use cases are flawed and partially missing with mistakes in them and not maintained for changes, so I was looking for "Deriving test cases from use cases is probably a bad idea". What does it even mean? Derive them from use cases? You mean do the use cases and see what happens? Well then that's all of them isn't it. Someone please tell me I'm wrong, I really want to know why A is correct and I'm wrong.

Thinking about it, likelihood is a very hard thing to calculate. We'd need to know a statistically significant number of unbiased selected occurrences of such derivations and when it was helpful and when it was not helpful for each.

What a bloody stupid question.

Score: 12/23

24. Which of the below would be the best basis for fault attack testing?

a) Experience, defect and failure data, knowledge about software failures.  
b) Risk analysis performed at the beginning of the project.  
c) Use Cases derived from the business flows by domain experts.  
d) Expected results from comparison with an existing system.  

My answer: A?

ISTQB says: A

I don't know what fault attack testing is and I guessed A. I don't know why it's better than risk analysis or whatever but I know faults (something that allows a product to exhibit a problem) and failures (something the product does that we wish it wouldn't) are directly linked. Threats cause faults to show failures. So I guessed A.

Score: 13/24

25. Which of the following would be the best test approach when there are poor specifications and time pressures? 

a) Use Case Testing.  
b) Condition Coverage.  
c) Exploratory Testing.  
d) Path Testing.  

My answer: C

ISTQB says: C

C is the only approach listed. Which of these vehicles has the highest top speed? a) Banana, b) Spoon, c) Motorbike, d) Soil

Score: 14/25

26. Which one of the following techniques is structure-based?

a) Decision testing.  
b) Boundary value analysis.  
c) Equivalence partitioning.  
d) State transition testing.  

My answer: D?

ISTQB says: A

What does that even mean? Structure, to me, means code and hardware and sample data and packaging and paper documents - the building blocks of the product. So decision testing - well it's not based on structure. Boundary values... again, not based on structure. None of them are, I just got desperate and put D.

Score: 14/26

27. You have started specification-based testing of a program. It calculates the greatest common divisor (GCD) of two integers (A and B) greater than zero. 

calcGCD (A, B); 

The following test cases (TC) have been specified. 

INT_MAX: largest Integer 

Which test technique has been applied in order to determine test cases 1 through 6? 

a) Boundary value analysis.
b) State transition testing.
c) Equivalence partitioning.
d) Decision table testing.

My answer: A & C

ISTQB says: A

I have started specification-based testing have I? Okay, what else dwells within this dystopian hell? Looks boundary value-y. Also to separate the nature of the inputs they've been put in equivalence classes. But I suppose it depends on what the damn specifications contain, doesn't it?

Score: 14/27

28. Consider the following state transition diagram and test case table: 

Which of the following statements are TRUE?

A. The test case table exercises the shortest number of transitions.
B. The test case gives only the valid state transitions.
C. The test case gives only the invalid state transitions.
D. The test case exercises the longest number of transitions.

a) Only A is true; B, C and D are false.
b) Only B is true; A, C and D are false.
c) A and D are true; B, C are false.
d) Only C is true; A, B and D are false.

My answer: B

ISTQB says: B

"The test case gives only the valid state transitions". I'm guessing they mean the test case table. Let me just say that I don't really fully understand the nature of the question, I just guessed this to be correct.

Score: 15/28

Anyone else as exhausted as I am? If not, meet me next time for "Test Management"!