Testers, Developers and Coders

What's the difference between a tester and a developer?

I originally looked at testing and development and the people that do it, like this:

I then migrated to believe in an overlap. Testers can sometimes develop, and developers can sometimes test.

Then came an important realisation. That testers are developers, insofar as they are part of the development of the software. Hopefully a well-used and important part.

However, there's a secret truth about this diagram. The strange thing about testing is that it doesn't require any particular skill, it just requires some sense of purpose. The desire to find things out about the product. When a coder compiles their code and sees a compiler error they've arguably just performed some testing. They have learned something about the product and then they can use the information to act on it. When they check their own work locally to see if it seems to behave sensibly before they put it in a build they have done testing. Informal testing, exploring the product for problems. This necessarily happens, it can't be avoided, even if they try! So where does that leave us, if coders are testers? Well, in some places they've decided that if testing can be done by anyone (which is pretty much true), then why not fire all of the testers and re-name the "developers"? Test is dead! Except that it's not, it's just being done by someone else.

Consider coders doing testing like bleach. It's potentially dangerous, but can be used for good. Now consider a view of testing where "testing can be automated", and "acceptance criteria" are a suitable substitute for actual requirements and non-shallow testing, as an acid that, if brought close to a sensitive surface like software development, can eat away at it. Now if we mix the bleach of tester-coders with the acid of testing ignorance we get a cloud of chlorine gas, choking the life out of coders, testers, software users, and anyone else who comes close enough. It's not likely to kill anyone in small enough doses, which is why it's possible to carry on with fake or bad testing especially in a big company where there's enough room for the gas to dissipate a little.

That's where the test expert lives. We are wardens of the health and safety of the people who design, build and use software. We ensure that the bleach is used responsibly in a well-ventilated area, and we keep dangerous acids locked away in a cupboard. We do this by not just exclusively testing, or concentrating on testing, but being skilled and tenacious in our pursuit of the truth and making it palatable enough to be realised by those blinded by their fantasies about their software. Don't strive to have testing achieved, strive to achieve good testing.

Improving Your Test Language - Automation

We think in language, and we communicate using language. Well, we perform a set of string translations that give the affordance of some meaning to someone else. I often look to Michael Bolton for clarity in the way we speak and write about testing, and his recent TestBash video is directly on the subject. I thought, as I hadn't updated my blog in months, that I'd post a little about the disjointed way some people in the industry talk and write about testing and how you might avoid doing the same - or if you want to continue doing the same understand the pitfalls of the language you're using and what you might be communicating to other people. A fair warning: this won't be news to many of you.

Automation

There is no automated testing. Automated / Manual Testing is a long-standing and extremely adhesive set of terms, but they should still be treated with due care. Testing is a human performance and cannot (currently) be automated. Manual Testing really just means "testing with your hands". In terms of software testing it cannot mean "testing without tools", in the sense that tools are required to interact with the product. One might describe hands and eyes as tools, or the computer itself, the screen, the hardware, the operating system, peripherals, the browser, and so on. These are tools that help you (enable you) to test the software.

I think that to most people automation means having the computer do a check and report the findings (automatic checking). At some point someone needs to design the checks and the reports. Also at some point someone has to read the results, interpret them, and assign them meaning and significance. These are points of access for human interaction with the tools being used in an example of tool-assisted testing.

That point is important - tool-assisted testing. All software testing is tool-assisted to some degree - and when one realises this it no longer splits testing neatly into two boxes where one is manual testing (executing test cases and some of "The Exploratory Testing") and the other is automated testing (a suite of automatic checks). Unfortunately it's a bit more complex and complicated than that. We need to look at testing as humans trying to find information. From that starting point we introduce tools because their benefits outweigh their costs. Well-written and maintainable automatic check tools can be really useful, but we must remember their costs - not just in terms of up-front financial costs, time costs, opportunity costs, maintenance costs, training costs and so on, but because a tool is an abstraction layer between the user and the software that introduces abstraction leaks. Automation (automatic checking) does not "notice" things it hasn't been strictly programmed to notice. Nor does it use the same interface to the software as a human does (although hopefully it's similar in ways that matter). Nor can it find a problem and deep-dive - meaning that it cannot interpret a false positive or false negative result. Nor can it assess risk and make meaningful decisions. Nor much of anything a tester does. Nor can it do much that a tester cannot do with "manual tools" (tools that aren't automatic check execution systems) given enough time. An automatic check suite "passing" just means that the coded checks didn't report a failure, not that suitable coverage was achieved.

This should empower you. It breaks a dichotomy of monolithic terms into a rainbow of possibilities. This should give you free reign to consider tools in your testing to make your testing more powerful, more fun, more efficient, even make testing possible... at a cost that you've thought about. Some of these costs can be reduced in the same way that testing itself can be improved - modularity and simpleness. Choosing simple, small or established tools like PerlClip, LICEcap, logs and log alerting systems, and one of my favourites: Excel can give you flexibility and power. You can pick the tools that suit your context - your favourite tools for your testing will differ greatly to mine, and for each time you test. I might use Fiddler today, Wireshark tomorrow, and Excel the next.

You don't have to choose between "Automated Testing" and "Manual Testing". They don't really exist. Put yourself in charge of your own testing and leverage the tools that get you what you want. Remember, though, that the tools you've chosen to help your processes will affect the processes you choose.

The Paucity of Pass/Fail

Pass and Fail seem to be ubiquitous terms in the testing industry. "Did the test pass?" and "How many failures?" seem to be innocent questions. But what, exactly, does pass or fail mean?

What does "pass" mean to you?

Think about that question for a second, before you move on.

Pass/Fail Reports

Checks

Let's say we're talking about a check done by a tool (sometimes called "automation"). We look at the results of that check, and it reports "PASS". What does that mean?

Let's look at this:

That's the result of a custom computer program I wrote for this automation example.

I think most people would be forgiven for thinking that this means that there is a test that's been coded that checks the login page to with a wrong password to ensure that it doesn't let the user into the system, and it passed. Most people would be forgiven for then thinking "well, we've testing logging in with an incorrect password".

If you (yes you) are a tester, then you are not the "most people" I was referring to. You should be highly suspicious of this.

What does pass mean to you now?

Tests

Let's look at this:

Tester	Test	Result
Chris	Logging in with the wrong password	PASS

This is a report given by me, a tester, for the sake of this example. I'll tell you that a human wrote all of this. Look at that PASS. Surely anything that green couldn't be wrong? If you were most people, you could be forgiven for thinking that the "logging in with the wrong password" test has passed. The coverage has been achieved. The testing has been done.

If you (yes, still you) are a tester, then you are not most people, and you should be highly suspicious of this.

So What?

Okay, testers, let's focus on those passes.

Checks

What does pass mean to you now?

To most people "pass" often means "as far as that test is concerned, everything's okay". Sometimes it means "that's been fully tested and is okay". Often it means "we don't have to worry about that thing any more".

But let's take a closer look. That automated test suite I showed you, the one with one test in it? It says that the "Invalid Login - Wrong Password" test passed. But it's not a full test, it's a check. What is it actually doing? I mean what does the "Invalid Login - Wrong Password" test do?

Well we know what it doesn't do. It doesn't investigate problems, evaluate risk, take context into consideration, interpret implicature, or do anything except what it was told to do. Maybe we investigate further and find out from the developer that what it does is enter an invalid login by having the computer enter "test1" into the username field and "password123" (which isn't test1's password) into the password field. Let's say that if, after clicking the "Login" button, the "Invalid Password" text appears, then it reports a pass, otherwise it reports a fail.

What does pass mean to you now?

Well, the explanation of the code means that the code that checked it returned a particular value (PASS) based on a specific check of a specific fact at a specific time on a specific platform, on this occasion.

Can we still have that happy feeling of "coverage" or "not having to worry" or "the testing being done"? Well, of course not. Here are some other things that would cause the code to return a "PASS" value and invalidate the test:

• The test data is wrong, or failed to load properly, and doesn't include a test1 user at all
• The "Invalid Password" text ALWAYS appears for any password
• The "Invalid Password" text appears for every first attempted login
• The text "Invalid Password" is hidden on the screen, but the checking system finds it in the DOM and reports it as found
• The text that appears for a valid password entry has been copy-pasted and is also "Invalid Password"
• The text "Invalid Password" appears elsewhere on the page after a valid login

These are scenarios that are isomorphic to the check's observed behaviour. That is to say that what the check "sees" appears the same in all of these cases, meaning that the check doesn't actually check for a wrong password login, it only checks for specific text after a specific event on a system with an unknown platform and data.

This means that for all of these cases the check reported a pass and there's a serious problem with the functionality.

We might say think because a computer said "pass" there is no problem. However, it might be that there are problems the check is not coded to do, or there may be a problem that the check does not describe because it's badly written, or something unexpected happened.

What does pass mean to you now?

Here's the actual real Ruby code I wrote for this automation example:

1 2 3 4 5 6

puts "Running Test Suite..." puts "" puts "Test: 'Invalid Login - Wrong Password' => PASS" puts "" puts "1/1 Tests run. 1 passes, 0 failures." puts "\n\n\n\n"

What does pass mean to you now?

Tests

Okay let's move on to that tester's report. A tester did the testing this time, and it passed! But while testers are intelligent and computers are not, they are frequently less predictable. What did the tester ACTUALLY do? Well, let's ask them.

Well, let's say that they said that they first checked the test data for a test1 user, and tried a valid login to confirm this. The system displayed "valid password" and gave them access.

Let's say that they said that they logged out then tried a bad password and found that it prevented them from logging in, and gave a useful, expected error message.

Let's say that they said that they tried to repeat the test a few times, and found the same behaviour.

What does pass mean to you now?

Feel better? I kind of feel better. But I think we know enough to question this, by now. What scenarios can you think of where the testing didn't find a problem related to this simple test idea? Here's a few that represent false positives or false negatives:

• The message doesn't display (or some other problem) on certain browsers
• The message doesn't display if you click the Login button twice
• When trying to log in with different passwords the system was just presenting the same screen instead of trying to log in
• The tester is testing an old version of the software that works, but the functionality is broken in the latest version.

Of more interest here are some that represent problems that weren't found:

• Every time the tester fails to log in it increments a "failed logins" value in the database. It's stored as a byte, so when it reaches 255 it throws a database error.
• The value mentioned above is responsible for locking out the user after 10 tries, so after 10 tries it ALWAYS displays "Invalid Login" on the screen, even with a valid login.

It's a fun experiment thinking of all the ways the tester didn't find existing problems while reporting a pass.

Guess what I (the tester) actually did to test the login page? That's right, nothing. I made it up. You should never have trusted me.

And what wasn't I testing? What system was I trying to test? How important is it that this works? Is it a small website to sell shoes, or a government site that needs to withstand foreign attacks?

A Passing Interlude

So let's review our meaning of "pass".

It seems to give the impression of confidence, coverage and a lack of problems.

It should give the impression of no found problems - which by itself is of exceedingly little value. Unless you know what happened and why that's important as far as "no found problems" is concerned you can't tell the difference between "good coverage and risk assessment" and "I didn't actually do any testing". Remember my test report, and my big green PASS? I didn't do any testing. The "PASS" by itself has no value. A non-tester might try one happy-path test and write PASS on their report having done no real investigation of the system.

If you're interested in a better way to report your testing then I recommend this Michael Bolton post as a jumping off point.

"PASS" is closer to meaning "Whatever we did to whatever we did it to, with however much we understand the system and however much we understand what it's supposed to do and whatever capability we have to detect problems we did not, on this occasion, with this version of this software on this platform find any problems."

I've focused on "pass" here, to show how weak a consideration it can be, and how much complexity it can obscure, but I'm going to leave it as homework to consider your version of "fail". What does "fail" mean to you? Do you use it as a jumping off point for investigations? Why don't you use "pass" as a similar jumping off point? How are your coded checks written - hard to pass or hard to fail? Why?

What Do I Do?

Remember what you're interested in. Don't chase the result of checks, chase down potential problems. Investigate, consider risk, empathise with users and learn the product. Consider problems for humans, not pass/fail on a test report. Use your sense and skill (and ethics) to avoid the pitfalls of dangerous miscommunication.

One of our jobs as testers is to dispel illusions people have about the software. People have illusions about the software because of various levels of fear and confidence about the product. Let's not create false fear and false confidence in our reporting - understand what pass and fail really are and communicate your testing responsibly.

What does pass mean to you now?