The Motion Sickness Heuristic is a way of combining salient elements of coverage with path testing in parts of systems that permit backtracking through the user workflow.
You may have heard of the Steeplechase Heuristic (from James Bach), where extreme data can test the limits of one input, but there may be other limits further downstream so it's a good idea to take the upper limit of an input and carry it through the workflow.
Well, imagine you're testing a system that has loops, or backwards flows. And don't forget that a browser has a "back" button. The Motion Sickness Heuristic means moving backwards and forwards and round loops until the system throws up a problem.
One can use the Motion Sickness Heuristic to provide the system with extreme or unusual inputs, and move back and forth through the system. It can be used to help learning by uncovering hidden paths/branches in path/branch coverage. It can be used to help uncover unusual behaviour while doing data testing. I use it to remember to go backwards as well as forwards in wizards and to test persistence and processing of data on unused branches. It's also useful to try to violate mandatory input constraints.
It challenges the assumption that users will always move forwards through the workflow, and never go back to change a mistake or to change their mind based on what they see in the current screen. It helps to test that context-sensitive information is displayed correctly in future screens when their trigger is changed. It helps to investigate what happens to data on branches that are subsequently not used.
Have fun!
Wednesday, January 29, 2014
Tuesday, November 26, 2013
Post Festum Code Changes
Just a quick one.
The mind of a programmer is rife with models of states and UI and objects and relationships. When they're coding something. Then later they're coding something else with different models of states and UI and objects and relationship. Ask them to go back and fix a bug in the first thing they wrote and they will do so from a fresh perspective. Ask them to go back and add functionality to old code and they'll do that with a fresh perspective.
Here's an example. The administration page has been developed to add users to a system. When it was developed it was after absorbing the relevant information, which included the fact that the user can choose an employee ID for the user and type it in - the system complains about any duplicates, preventing a user from being saved. To force the choice to NOT have an employee ID the default is 00000 and must be removed manually or updated to something new. The system seemed to be working great, then it was discovered that the customer wants a way to add lots of users from a list of first names and last names. This was added, but the person adding it forgot about the requirement for users to have unique (but optional) employee IDs and the code that turned the list into the user injected the 00000 default employee ID into all of the users, bypassing the input validation!
This is what I'm now calling a "Post Festum" code change. This is from the Latin usually meaning "too late" or "after the fact" - as in too much time has passed since the original coding. Helpfully, it literally means "after the feast", as in the main party is already over.
The mind of a programmer is rife with models of states and UI and objects and relationships. When they're coding something. Then later they're coding something else with different models of states and UI and objects and relationship. Ask them to go back and fix a bug in the first thing they wrote and they will do so from a fresh perspective. Ask them to go back and add functionality to old code and they'll do that with a fresh perspective.
Here's an example. The administration page has been developed to add users to a system. When it was developed it was after absorbing the relevant information, which included the fact that the user can choose an employee ID for the user and type it in - the system complains about any duplicates, preventing a user from being saved. To force the choice to NOT have an employee ID the default is 00000 and must be removed manually or updated to something new. The system seemed to be working great, then it was discovered that the customer wants a way to add lots of users from a list of first names and last names. This was added, but the person adding it forgot about the requirement for users to have unique (but optional) employee IDs and the code that turned the list into the user injected the 00000 default employee ID into all of the users, bypassing the input validation!
This is what I'm now calling a "Post Festum" code change. This is from the Latin usually meaning "too late" or "after the fact" - as in too much time has passed since the original coding. Helpfully, it literally means "after the feast", as in the main party is already over.
Tuesday, September 10, 2013
Scripts Part I - What is a script?
Let's Define!
My current definition of a script is:A set of instructions made explicit by one agent such that another (or the same) agent can later interpret them in order to replicate a process to achieve a desired outcome or behaviour.
So a script is, essentially, a communication tool. It can be verbal or written, it can be instructions to follow in order or a set of unordered instructions, but it is always done with the intention that the instructions will be followed by someone or something.
Here are some examples of scripts:
A computer program
A food recipe
A shopping list
A checklist
Communication
Implicit information in communication
When humans talk to each other we express two elements of communication. In pragmatics they are known as explicature and implicature. Explicature is what is actually said; for example "We went shopping, had a meal and ended up in the cinema". Implicature is what is implied by what is said. In the above example one might assume that going shopping, having a meal and ending up in a cinema happened in that order, but the sentence remains logically true even if the order is changed. There's also an implication that going to the cinema involved watching a film as part of a day out but the sentence remains logically true if the group stepped into a cinema then turned around and immediately walked out.These two elements of communication are important when considering the audience for a script - the script has to be interpreted in order to be followed and that's what will affect the outcome of behaviour that actually happens (as opposed to the one desired by the script author)
Computers vs People
A computer program is a kind of script. It's a set of instructions made explicit by a programmer that the computer follows to execute the intentions of the programmer. Computers only pay attention to explicature in script instructions. This can be translated into the more common phrase "computers only do what you tell them to do".Computer scripts may fail because computers are very picky about their explicature (they must be syntactically accurate). It may fail because the computer will execute the script as it exists while ignoring implicit desires. Just because you write a comment in the script saying "this function will return the square root of a number" does not mean it will. Just because you used a variable name called "square_root_of_input" does not mean that's what the variable contains - and your computer cannot care. We write unit tests in code as a way to try to ensure that the behaviour of the code we write matches our intended purpose - those are the lengths we go to to compensate for how mindless computers are and how literally they interpret our scripts.
Computers ignore implicature. They ignore context. And this is why they're so difficult to communicate with - human language is rife with implicature. We put up with this communications issue because computers are able to perform tasks (even ones that we are bad at or find impossible) in volume at an enormous rate.
A food recipe is a kind of script. It is a set of instructions made explicit by a person so that a (often) different person can follow them in order to end up with similar food. People have many powers that computers lack. They can understand implicit and tacit information. A recipe may fail because of a misunderstanding concerning implicit information. In the UK "crumble 3 plain biscuits" probably means McVities Digestives. In the US "biscuits" are basically a small bread a bit like a nearly tasteless scone. Try making a biscuit base with those.
Because it's followed by a person a recipe can have certain implicit or tacit information in it. For example if I'm told to use "self-raising flour" and I don't have any perhaps I might use plain flour and a leavening agent. The explicature is "add 200g self-raising flour". The implicature is "add something that has similar properties of 200g self-raising flour in the context of baking the pie this recipe describes". I know I can probably skip the "Add almonds" step but I can't very well skip "Bake at 180C for 40 minutes" because I can guess how differently they will affect the quality of the end product. This is the power humans have over computers.
Scripts in Testing
Automated Testing
Computers will interpret only the literal truth of explicature in scripts. This is the essence of checking to me - ignorance of implicature. This is a price we pay in exchange for speed - computers are much faster at making these checks.Manual Scripts
Humans will interpret the implicature of a script. Humans are UNABLE to ignore implicature, we are simply not wired for it. It's part of our emotions. It's how we can process information in a heuristic way - not always accurately but with the efficiency to be evolutionarily successful. It's how we communicate complex ideas and their impact. It's what allows us to create art.This means we cannot perform checks (under the strictest definition of check). A human check, I suppose, is the absence of knowledge to guide implicature in specific observations while making an evaluation in a product - we still make certain natural inferences but we can try to ignore parts of the context or try to behave as if we don't know the context in order to make a simple confirmation of the truth value of the explicature of an instruction (or what would be the explicature of an instruction were it uttered).
But why try to ignore inferences when we can use them? Humans have enormous power over computers to investigate, learn, observe, interpret, infer, follow up, ask for information. So why write scripts to tell them what to do? If they're powerful enough to investigate a product surely it's cheaper and easier just to train them on how to look for problems rather than explicitly tell them what problems to look for. There is an answer to the above question which I hope to get to in a separate post.
What Is/Is Not A Script (In Testing)?
Things That Are Scripts (In Testing)
Automated Check Scripts
Automated "tests" are a set of instructions (the code) made explict by one agent (coded) such that another agent (the computer/compiler/interpretter) can later interpret (run/compile/interpret) them in order to replicate a process (making many checks quickly and forming some kind of output with the data) to achieve a desired outcome (the results of specific observations of the system in a way that a human can easily interpret, understand and evaluate)
Things That Aren't Scripts (In Testing)
Notes
Notes are not usually scripts. Notes are simply anything made explicit for the purposes of later reference. Scripts more specific - they refer to a set of instructions to be later interpreted to achieve a desired behaviour or outcome. Not all notes consist of a set of instructions and not all notes are designed to achieve a desired behaviour or outcome. All scripts are notes. Not all notes are scripts - in fact so few that we can make a abductive leap and say that when someone says "notes" they don't mean "scripts" because if they did they'd likely have said "scripts" to differentiate it from "notes".Missions/Charters
A mission or charter is similar to a script in that it contains some explicit information and is used to communicate something in order to achieve a desired outcome or behaviour. The difference is that the communicated information is not a list of instructions to replicate a process. The instructions are not made explicit - the desired outcome or behaviour is. The steps taken (the set of instructions that replicate the process) is left up to the agent. It could be that a mission or charter contains a checklist (a type of script) so missions and charters can use the power of scripts. It may note a list of things that must be tested, information that's especially important, instructions on how to configure the system for the testing, and so on.So while a recipe is a kind of script a note that says "We need a sponge cake. Plenty of lemon icing, please, and that fancy swirly decoration thing on the top." is a mission, or charter. The recipe is left up to the chef - or maybe the recipient of the note will go out and buy a cake, or commission a custom cake.
Things That Might Be Scripts (In Testing)
Checklists
A checklist is simply a list of checks. The problem here is that it depends on how much information is left implicit in that list of checks that will determine if the checklist matches my definition of a script. Remember, a script is "a set of instructions made explicit by one agent such that another (or the same) agent can later interpret them in order to replicate a process to achieve a desired outcome or behaviour". This means that if the checklist has the intention and capacity to replicate a process then it's a script. If the process is left to the agent interpreting the checklist to achieve the desired outcome or behaviour then it's not. I'd go as far to say that most checklists are not scripts.
I'll update the list above as I find more things that are confused with scripts.
Thanks for reading, now please disagree with me!
- Kinofrost
Monday, April 8, 2013
Bad Testing Definitions
Life has become busy. I'm giving presentations and talks, I've taken up morning running, I'm getting involved with local groups and so on. As a result I haven't had any time to blog, tweet or otherwise social media anything. So... here's my list of previously-tweeted "Bad Testing Definitions"!
#1: Focus/Defocus - what your eyes do after staring at the same page of the interface for an hour
#2: Exploratory Testing - Taking your work laptop with you when you go camping
#3: Risk-based testing - deciding your next test action using test techniques on post-it notes and a dartboard
#4: Test Coverage - what you wear while testing
#5: Test Cases - the capitalization of your text-based test data
#6: White-Box Testing - testing light-coloured modal windows in the UI
#7: Stress Testing - all testing up to and including 1 week before release
#8: Cross-Browser Testing - When you’re web testing and the browser throws a tantrum
#9: Touring - Moving companies once every 6 months
#10: Risk Impact - What you feel when you realise you didn’t test something important
#11: Claims Testing - Tests performed on functionality a customer has complained about
#12: Performance Testing - Making a song and dance about reports and metrics instead of doing any testing
#13: Test vs Check - How much testing you do for what you get paid each month
#14: Regression Testing - Testing while invoking the knowledge and experience of past lives
#15: Model-based testing - Fashion show focus groups
#16: Acceptance Testing - coming to terms with the state of the software
#17: Alpha Testing - Winning arguments online about testing term definitions
#18: User Acceptance Testing - confirming receipt of payment from customers
#19: Conformance Testing - Telling the project manager what they want to hear
#20: Volume Testing - Testing while sitting near to development on a Friday afternoon#21: Sanity Testing - Installing the fifth broken build in a row
#22: Fuzzy Penetration Testing - Some of these write themselves.#23: End to End Testing - Eating at a new Indian restauran
#24: Agile Testing - Dodging questions about the state of the product
#25: Best Practice
#26: Exhaustive Testing - Running out of coffee
#27: Black Box Testing - Using electronic devices while on a flight
#28: Static Testing - Touching the product whilst wearing a polycotton onesy
#29: All-Pairs Testing - The male component of a test team
#30: Backward Compatibility Testing - Idiot-proofing
#1: Focus/Defocus - what your eyes do after staring at the same page of the interface for an hour
#2: Exploratory Testing - Taking your work laptop with you when you go camping
#3: Risk-based testing - deciding your next test action using test techniques on post-it notes and a dartboard
#4: Test Coverage - what you wear while testing
#5: Test Cases - the capitalization of your text-based test data
#6: White-Box Testing - testing light-coloured modal windows in the UI
#7: Stress Testing - all testing up to and including 1 week before release
#8: Cross-Browser Testing - When you’re web testing and the browser throws a tantrum
#9: Touring - Moving companies once every 6 months
#10: Risk Impact - What you feel when you realise you didn’t test something important
#11: Claims Testing - Tests performed on functionality a customer has complained about
#12: Performance Testing - Making a song and dance about reports and metrics instead of doing any testing
#13: Test vs Check - How much testing you do for what you get paid each month
#14: Regression Testing - Testing while invoking the knowledge and experience of past lives
#15: Model-based testing - Fashion show focus groups
#16: Acceptance Testing - coming to terms with the state of the software
#17: Alpha Testing - Winning arguments online about testing term definitions
#18: User Acceptance Testing - confirming receipt of payment from customers
#19: Conformance Testing - Telling the project manager what they want to hear
#20: Volume Testing - Testing while sitting near to development on a Friday afternoon#21: Sanity Testing - Installing the fifth broken build in a row
#22: Fuzzy Penetration Testing - Some of these write themselves.#23: End to End Testing - Eating at a new Indian restauran
#24: Agile Testing - Dodging questions about the state of the product
#25: Best Practice
#26: Exhaustive Testing - Running out of coffee
#27: Black Box Testing - Using electronic devices while on a flight
#28: Static Testing - Touching the product whilst wearing a polycotton onesy
#29: All-Pairs Testing - The male component of a test team
#30: Backward Compatibility Testing - Idiot-proofing
Tuesday, December 18, 2012
Mysteries and Puzzles
National security expert Gregory Treverton (2009) wrote for Smithsonian Magazine in 2007 the following:
"Even when you can't find the right answer, you know it exists. Puzzles can be solved; they have answers.
But a mystery offers no such comfort. It poses a question that has no definitive answer because the answer is contingent; it depends on a future interaction of many factors, known and unknown."
The article is worth reading at some length if you haven't already done so (see references).
And I think within this definition lies a great deal to say about attitudes towards software testing. You understand the infinite test space and the necessary uncertainty of the quality of the systems you are testing, but I believe a lot of confusion, from certain people who are usually in non-testing roles, stems from their misunderstanding of the difference between determining good perceived quality through testing as puzzle solving and as mystery investigation.
The essential differentiating factor, as I see it, between a puzzle and a mystery is it's relationship to the quantity of information at hand. An unsolvable puzzle is one that simply requires more information to be added. Or as Malcolm Gladwell (2009, pp.160) puts it in his book What the Dog Saw: "A puzzle grows simpler with the addition of each new piece of information". A mystery cannot be solved purely through the addition of more information - in fact as we increase focus on collection of information then any further information we gain will contain a majority of noise in which more problems can hide; making the issue not necessarily worse, but necessarily bigger.
A bug found in production by a user is a puzzle; We can identify the problem, and immediately see what the desired behaviour is. We can trace it back, identify its root cause (or near root cause) and solve it. Preventing bugs going into production (basically: software testing) is a mystery; it's a cognitive process of questioning and re-framing with no (as yet) known problems and confusion over desired behaviour. We have nothing from which to trace back and we're executing the root causes (or near root causes) for problems that we haven't found yet.
This is one reason that we can only make predictive insights into user-experienced software quality; true quality is un-knowable. We have an understandable obsessive focus with solutions in business, and it's easy to see why, but treating software testing as a solution is risky if we identify the wrong problem to be solved. Our biggest problem is not a lack of information about a product, but too much. We have documentation, implementations, previous experience, relationships to the product, relationships to the software provider and its employees, the structure and culture of the company, the users, the customer, and I'm sure you can think of more. The problem we have is the attempt to identify what is useful, questioning both the product and the project. It's not about learning and applying a requirements document (like some developers do), it's about understanding and evaluating it; interpreting data into information and refining information into heuristic oracles. The investigation into the mystery of software bugs. To be better at software testing it isn't usually more initial information we need, but better analysis of the information we have (meaning we gather better information, not just more).
This is exactly why testing is a human-centric process and why it requires application of thought and engagement. As Gladwell (2009, pp.166) states: "Puzzles are 'transmitter-dependent'; they turn on what we are told. Mysteries are 'receiver-dependent'; they turn on the skills of the listener". And it's our skills in challenging what we are told, and more generally as information listeners, that make us good software testers. We are in the practice of a kind of Bayesian analysis of information, adjusting our belief in the subjective to rationally account for new evidence.
The logical solution to the puzzle of software testing is simply the collusion of information collection and persistence. The mystery of software testing requires experience, insight and investigation. Puzzles require us to think. Mysteries require us also to think about the thinking. This is why bug fix verification is easy, but boring. It's no longer a mystery to solve, but a set of puzzles... but if we continue to apply the mystery of software testing through Rapid and Exploratory techniques such as "branching and backtracking" (Bach, J., 20xx, pp.20) areas of opportunity then when we do bug fix verification we immediately add value to the testing in higher risk (due to the necessary recent code changes to fix the bug) areas.
Business has become good at solving puzzles, but seems to ignore or avoid mysteries; and as Philip Herr (2011) says: "And if we have been especially successful at solving puzzles, we may be tempted to define all business problems as such. (As the saying goes, “If all you have is a hammer, everything begins to look like a nail.”).". Our over-reliance on problem solving skills in business, combined with a desire to demarcate everything into metrics and simple, usable facts, has lead to a misunderstanding of the complexities and puzzle-resistant elements of software testing and therefore how testing can be best used to add value to any project. If you've ever worked in a company that views testing as a "necessary evil" or an exercise in compliance then you'll understand the point I'm making.
Dealing with useful information on and around a product being mixed together with a greater quantities of useless information, misleading information, incorrect information, and observation posing as objective fact requires us to look at finding important bugs quickly as a mystery, not a puzzle. As Gladwell (2009, pp.168-169) says, "If you can't find the truth in a mystery - even a mystery shrouded in propaganda - it's not just the fault of the propagandist. It's your fault as well."
References:
Bach, J., 2006. Breaking Down (and building up) Exploratory Testing Skill. [online] Available at: <http://www.quardev.com/content/whitepapers/ET_dynamics_Google_3_06.pdf> [Accessed 16 December 2012]
Gladwell, M., 2009. What the Dog Saw: and Other Adventures. [e-book]. Penguin UK. Available at: play.google.co.uk <http://play.google.co.uk> [Accessed 16 December 2012].
Herr, P., 2011. Solving Puzzles Delivers Answers; Solving Mysteries Delivers Insights. Brand News, [online] Available at: <http://www.millwardbrown.com/Libraries/MB_POV_Downloads/MillwardBrown_POV_Solving_Puzzles_Delivers_Answers.sflb.ashx> [Accessed 16 December 2012]
Tiverton, G., 2007. Risks and Riddles. Smithsonian Magazine, [online] Available at: <http://www.smithsonianmag.com/people-places/presence_puzzle.html> [Accessed 16 December 2012]
"Even when you can't find the right answer, you know it exists. Puzzles can be solved; they have answers.
But a mystery offers no such comfort. It poses a question that has no definitive answer because the answer is contingent; it depends on a future interaction of many factors, known and unknown."
The article is worth reading at some length if you haven't already done so (see references).
And I think within this definition lies a great deal to say about attitudes towards software testing. You understand the infinite test space and the necessary uncertainty of the quality of the systems you are testing, but I believe a lot of confusion, from certain people who are usually in non-testing roles, stems from their misunderstanding of the difference between determining good perceived quality through testing as puzzle solving and as mystery investigation.
The essential differentiating factor, as I see it, between a puzzle and a mystery is it's relationship to the quantity of information at hand. An unsolvable puzzle is one that simply requires more information to be added. Or as Malcolm Gladwell (2009, pp.160) puts it in his book What the Dog Saw: "A puzzle grows simpler with the addition of each new piece of information". A mystery cannot be solved purely through the addition of more information - in fact as we increase focus on collection of information then any further information we gain will contain a majority of noise in which more problems can hide; making the issue not necessarily worse, but necessarily bigger.
A bug found in production by a user is a puzzle; We can identify the problem, and immediately see what the desired behaviour is. We can trace it back, identify its root cause (or near root cause) and solve it. Preventing bugs going into production (basically: software testing) is a mystery; it's a cognitive process of questioning and re-framing with no (as yet) known problems and confusion over desired behaviour. We have nothing from which to trace back and we're executing the root causes (or near root causes) for problems that we haven't found yet.
This is one reason that we can only make predictive insights into user-experienced software quality; true quality is un-knowable. We have an understandable obsessive focus with solutions in business, and it's easy to see why, but treating software testing as a solution is risky if we identify the wrong problem to be solved. Our biggest problem is not a lack of information about a product, but too much. We have documentation, implementations, previous experience, relationships to the product, relationships to the software provider and its employees, the structure and culture of the company, the users, the customer, and I'm sure you can think of more. The problem we have is the attempt to identify what is useful, questioning both the product and the project. It's not about learning and applying a requirements document (like some developers do), it's about understanding and evaluating it; interpreting data into information and refining information into heuristic oracles. The investigation into the mystery of software bugs. To be better at software testing it isn't usually more initial information we need, but better analysis of the information we have (meaning we gather better information, not just more).
This is exactly why testing is a human-centric process and why it requires application of thought and engagement. As Gladwell (2009, pp.166) states: "Puzzles are 'transmitter-dependent'; they turn on what we are told. Mysteries are 'receiver-dependent'; they turn on the skills of the listener". And it's our skills in challenging what we are told, and more generally as information listeners, that make us good software testers. We are in the practice of a kind of Bayesian analysis of information, adjusting our belief in the subjective to rationally account for new evidence.
The logical solution to the puzzle of software testing is simply the collusion of information collection and persistence. The mystery of software testing requires experience, insight and investigation. Puzzles require us to think. Mysteries require us also to think about the thinking. This is why bug fix verification is easy, but boring. It's no longer a mystery to solve, but a set of puzzles... but if we continue to apply the mystery of software testing through Rapid and Exploratory techniques such as "branching and backtracking" (Bach, J., 20xx, pp.20) areas of opportunity then when we do bug fix verification we immediately add value to the testing in higher risk (due to the necessary recent code changes to fix the bug) areas.
Business has become good at solving puzzles, but seems to ignore or avoid mysteries; and as Philip Herr (2011) says: "And if we have been especially successful at solving puzzles, we may be tempted to define all business problems as such. (As the saying goes, “If all you have is a hammer, everything begins to look like a nail.”).". Our over-reliance on problem solving skills in business, combined with a desire to demarcate everything into metrics and simple, usable facts, has lead to a misunderstanding of the complexities and puzzle-resistant elements of software testing and therefore how testing can be best used to add value to any project. If you've ever worked in a company that views testing as a "necessary evil" or an exercise in compliance then you'll understand the point I'm making.
Dealing with useful information on and around a product being mixed together with a greater quantities of useless information, misleading information, incorrect information, and observation posing as objective fact requires us to look at finding important bugs quickly as a mystery, not a puzzle. As Gladwell (2009, pp.168-169) says, "If you can't find the truth in a mystery - even a mystery shrouded in propaganda - it's not just the fault of the propagandist. It's your fault as well."
References:
Bach, J., 2006. Breaking Down (and building up) Exploratory Testing Skill. [online] Available at: <http://www.quardev.com/content/whitepapers/ET_dynamics_Google_3_06.pdf> [Accessed 16 December 2012]
Gladwell, M., 2009. What the Dog Saw: and Other Adventures. [e-book]. Penguin UK. Available at: play.google.co.uk <http://play.google.co.uk> [Accessed 16 December 2012].
Herr, P., 2011. Solving Puzzles Delivers Answers; Solving Mysteries Delivers Insights. Brand News, [online] Available at: <http://www.millwardbrown.com/Libraries/MB_POV_Downloads/MillwardBrown_POV_Solving_Puzzles_Delivers_Answers.sflb.ashx> [Accessed 16 December 2012]
Tiverton, G., 2007. Risks and Riddles. Smithsonian Magazine, [online] Available at: <http://www.smithsonianmag.com/people-places/presence_puzzle.html> [Accessed 16 December 2012]
Sunday, November 18, 2012
Testing Towers
I typed into Google the following:
"We test software to"
And read the results. I took some ideas from these results, and some of my own, and started to write down reasons to test. Things that motivate testers. Then I put them in an ordered list... looking at the list now it looks like I put them in (arguably) order of lowest to highest risk, the most important (to some imaginary business developing the software) at the bottom and what is considered the "icing on the cake" at the top. I looked for priority, and I looked for context, and I came up with a lot of different building blocks. Here are some:
Good User Experience (product is easy and/or fun to use, even if it doesn't do what is required)
Product Solves Problem (product solves the problem it was set out to solve, in that the functionality it has actually fixes the problem that existed that the product was made to fix)
Product Has Fewest Number Of Bugs (the highest possible number of bugs has been found, leaving the fewest number remaining in the software)
Product Won't Do Bad Things In Production (the product doesn't do anything that would
Product Is Reliable (product doesn't crash, shut down, run slowly, fail)
Product Won't Kill People (doesn't fail, causing death or serious injury)
Product "Works" (is operational. The buttons work, the functions do something)
I'm Learning (I know nothing directly about the "quality" of the product as a result, but I know more about the product, so I understand better what a "quality" product would mean)
Manager Isn't Bothering Me (ignore the software, I have my manager off my back)
Fulfils Requirements Doc (does what the documentation says it should)
Product Is Testable (product can actually be tested)
Based on context (software development methodology, culture, strategy, resources, and a host of other things) these could be stacked in multiple ways. You might want "product won't kill people" in there specifically because you're testing a safety-critical product. But if you're testing military weaponry perhaps you want your product to be able to kill people. It could be that Product Is Testable is on the bottom as it's required to test that it Fulfils Requirements in documentation and has the Fewest Number of Bugs which thus Solves The Problem. Personally I think that would be appalling in all sorts of ways, but it might be what your testing tower looks like. Like this:
| Solves Problem | ||
| Fewest Bugs | Fulfils Requirements Doc | |
| Product Testable | ||
Unfortunately I'd say that the Fewest Bugs block doesn't really exist. I'd say that the idea of finding a quantum quantity of bugs in an infinite test space is nonsensical. *POOF* it's gone, and the Solves Problem brick is unstable and could fall at any second! I'd say also that Fulfilling Requirements Doc does not support the solution of the test problem, but only the solution of the requirements doc, which is not the same thing. *CRACK*, it's broken. And the whole (if overly short for this example) testing tower comes tumbling down. Oops.
We all test to investigate the software, but what are you investigating? What is your aim? Something worthwhile I hope.
So, what are your bricks? Are they strong ones that support the layers above? Are they in the correct order, so as to stop your tower being top-heavy? Do they even exist in your tower (apply, and logically follow, in the real world)? It's an interesting way to see your processes, and the dependencies in your workflow, and what you're trying to achieve by testing the product, and for whom you are testing it. I wouldn't use it for modelling your desired practices, personally, but it does give some insight into these things, and possibly even the culture and practice of your company, and where things could be improved. Build your tower, and see if it has a worthwhile top to climb to. Then throw logic at it and see if it's still standing.
If you're looking for more (or better) bricks you might turn to something like James Bach's CRUSSPIC STMPL quality characteristics heuristics, or his CIDTESTD project environment heuristics. You could take stock of your resources, or learn about a software development methodology that you don't use.
What does your testing tower look like?
Edit:
Thinking about this recently I've used the blocks to identify the divergence from why we should test to why we DO test and the importance of company culture. Strategy, planning, learning... all vital to set out what we should do. Add a layer of culture, habit and company practices and you get what we ACTUALLY do.
Tuesday, November 13, 2012
Learn to Code or Learn to Stack Shelves
Okay, that was a little inflammatory. I'll calm it down.
I haven't read much on the subject of testers learning to code, but people seem to be in one of three camps. The "nearly all testers must be able to code" camp, the "nearly all testers don't need to be able to code" camp and the "I kinda sorta agree with both" camp, which is actually the second camp but without the strong sense of reaction to the first.
I think one problem is that there is a great deal of diversity in testing, including automation coders, developers who write unit tests, exploratory testers, script checkers... and let's face it not EVERYONE needs to learn to code. We're not dragging "Introduction to Java" books into every vaguely IT role and telling them they need to learn this stuff. I'm next to certain my managing director does not know how to code. And I would say that the variance in testers (and checkers) and the context in which they work, is so widely varied as to bear the weight of my hyperbolic example.
Let's try it this way. Ask yourself this question:
Would learning to code improve my odds on remaining employed and/or ability to test what I'm testing?
If "no", then don't learn to code. If "yes" then ask yourself this question:
Would learning to code be the best use of my time and energy it would take?
If "no", then don't learn to code. If "yes"... I'm sure you can work out the rest.
Firstly I must point out that if all you do is execute scripts then you are a checker, and you might be in trouble. Sorry.
I haven't read much on the subject of testers learning to code, but people seem to be in one of three camps. The "nearly all testers must be able to code" camp, the "nearly all testers don't need to be able to code" camp and the "I kinda sorta agree with both" camp, which is actually the second camp but without the strong sense of reaction to the first.
I think one problem is that there is a great deal of diversity in testing, including automation coders, developers who write unit tests, exploratory testers, script checkers... and let's face it not EVERYONE needs to learn to code. We're not dragging "Introduction to Java" books into every vaguely IT role and telling them they need to learn this stuff. I'm next to certain my managing director does not know how to code. And I would say that the variance in testers (and checkers) and the context in which they work, is so widely varied as to bear the weight of my hyperbolic example.
Let's try it this way. Ask yourself this question:
Would learning to code improve my odds on remaining employed and/or ability to test what I'm testing?
If "no", then don't learn to code. If "yes" then ask yourself this question:
Would learning to code be the best use of my time and energy it would take?
If "no", then don't learn to code. If "yes"... I'm sure you can work out the rest.
Coding vs Testing?
Okay, so it's a bit facetious, but you get the idea. There is no answer to "should testers learn to code", and anyone that says there is a universal answer is either being deliberately obtuse or has failed to understand the non-universal nature of testing. So I suppose the only thing now is to give some ideas to the benefits and drawbacks to learning to test.Firstly I must point out that if all you do is execute scripts then you are a checker, and you might be in trouble. Sorry.
What learning to code will cost you
Firstly the downsides.
1. Time
Learning to code takes time. That's about it. You could use that time
2. Effort
Learning to code takes energy and focus which could be better directed to learning your product, or learning more about testing or related subjects (goodness knows there's enough to keep you busy!)
3. Choice
Choosing to learn to code in one language does not mean that you are immediately familiar with others (although it does help). So choosing one language to code in will not be much use to a tester if you find yourself in a position that you can no longer use it.
4. You might be awful at it
The mind of a developer is a confusing place, as I'm sure you probably know, and this difference isn't accidental. Not everyone is good at coding. It can be tiring, frustrating, repetitive and boring. Some people love it, but I would venture that a great tester would not make for a great developer.
5. Coding can be dull
You're probably a tester, and you're interested in what's new and interesting. Different angles and approaches, alternative ways of thinking, exploration, creativity and drive. Personally I find coding at any sort of professional level my own personal idea of hell; the same syntax every single day, building non-physical creations with a virtual soupy bucket of strands that must be built, re-factored and honed to a solution. Not for me. Although I do know how to code in some languages, and I find it quite fun... but only knowing that I don't have to be exact or precise or impress anyone in particular. Just as I enjoy occasionally writing poetry but I'd be crying with frustration into my notepad if I were paid to do it every day for a critical audience... wow, I suddenly feel a wave of empathy with developers!
What learning to code will get you
1. Insight
I've found knowing how software is made (from a blank screen to a simple set of functions) helps me understand the mistakes and shortcuts that occur. I use this information to perform quick-attack tests. If you are in a position to apply creativity to your testing then knowing how the people that make the bugs make the bugs will give you an array of tools to help you find them.
2. Alternative employment
A dodgy claim, but knowing how to code gives you another trade to fall back on. To be quite honest if you have a lot of experience as a tester and no experience as a coder you will very likely have a lot of trouble getting this sort of work, but.. it might help.
3. Automation
Automated checks with any worth usually require some level of testing experience. Chances are there's something you could automate, or partially automate, to help you reach your testing objectives.
4. Tool building
I use my coding ability to build tools to make my job easier. I am a true computer user, in that I am essentially lazy. I spend a little time to save a lot of time in my testing, and reporting. For example, I've coded systems that detect the version and build of the system I am testing and copy bug reporting information to the clipboard because I'm too lazy to copy, paste and fill in a bug report's environment details. And I do that many times a day. Now it's never victim to copy/paste errors, and it saves me a lot of frustration and a bit of time.
Okay, so we broke that down into parts... Where do we go from here?
Well, if you feel that you'll get something out of learning to code, and that you have some passion for it, and the time and energy to spend, then go for it! I think it's vitally important that you have an interest in coding to be even vaguely successful at it. If I came to the belief that I needed to eat broccoli every day to keep my job I'd start looking for another job... even if I was very good at it without any need for broccoli. I do coding as a hobby some times. Mind you, I pick locks, play Dwarf Fortress and have Magic tournaments with my girlfriend, so maybe I'm not a yardstick of entertainment. Either way, I think the main point in all this that the phrase
"Testers need to learn to code"
or similar are rife with hidden presumptions and worrying overtones. The meaning of "Tester", the meaning of "need" and the meaning of "code". The variance of testers I've already covered, and I hope I've gotten across some of the needs (or wants!). Of course "code" means so many different things... and these (and other points) have been far better covered by Rob of "The Social Tester", here.
So will you learn to code? Or perhaps there's another book on epistemology, general systems or game theory that you have left unread on your shelf that would provide you more insight and use. I won't advocate any need to learn to code, but if you want to I'd be the last to tell you not to. Whatever you take away from this, don't forget to learn something every day.
Edit
I've come to realise that the "you need to learn to code" accusation is levelled at people whose testing value isn't sufficient as to keep them employed. Perhaps it's not the "testers" (checkers) that need to learn to code, but the higher-ups that need to learn how to get value of their testing, and therefore learning how to advocate good testing (and admonish bad testing) would be more valuable in keeping one's job than learning to code.
Edit
I've come to realise that the "you need to learn to code" accusation is levelled at people whose testing value isn't sufficient as to keep them employed. Perhaps it's not the "testers" (checkers) that need to learn to code, but the higher-ups that need to learn how to get value of their testing, and therefore learning how to advocate good testing (and admonish bad testing) would be more valuable in keeping one's job than learning to code.
Subscribe to:
Posts (Atom)