Thursday, June 19, 2014

I Failed The ISTQB Practice Exam - Part VI & Final Thoughts

Series: Part IPart IIPart III & IVPart V

I was on 19/36...

Part VI - Tool Support For Testing

37. From the list below, select the recommended principles for introducing a chosen test tool in an organization? 

1. Roll the tool out to the entire organization at the same time. 
2. Start with a pilot project. 
3. Adapt and improve processes to fit the use of the tool. 
4. Provide training and coaching for new users. 
5. Let each team decide their own standard ways of using the tool. 
6. Monitor that costs do not exceed initial acquisition cost. 
7. Gather lessons learned from all teams. 

a) 1, 2, 3, 5.
b) 1, 4, 6, 7.
c) 2, 3, 4, 7.
d) 3, 4, 5, 6.

My Answer: B - only one without 3, except 6!

Obviously 3 is stupid, so (b) must be the answer! Except that it also has 6, which is rediculous. 1 would vary depending on the tool (does the CEO need RapidReporter?), 2 might be important if it's a big tool, 3 means being shaped by our tools which is dangerous, 4 might be necessary if they haven't used the tool before and it's complex or we want to use it in a certain way, 5 is a possibility (I don't teach anyone how to use Excel or a log viewer in a specific way, for example), 6 seems weird and 7 seems like a good idea. I suppose if it wasn't for 3 I might have picked C.

And who's the guy recommending these principles? Come on, someone must have had to. Unless they're sent down from on high etched into stone tablets or something.

Score: 19/37

38. Which one of the following best describes a characteristic of a keyword-driven test execution tool?

a) A table with test input data, action words, and expected results, controls execution of the system under test
b) Actions of testers recorded in a script that is rerun several times.
c) Actions of testers recorded in a script that is run with several sets of test input data
d) The ability to log test results and compare them against the expected results, stored in a text file.

My Answer: A

Another point for yours truly as he deftly answers the challenge of keyword-driven test execution by choosing the only answer that contains any test execution.

Score: 20/38

39. Which of the following is NOT a goal of a Pilot Project for tool evaluation?

a) To evaluate how the tool fits with existing processes and practices.
b) To determine use, management, storage, and maintenance of the tool and test assets.
c) To assess whether the benefits will be achieved at reasonable cost.
d) To reduce the defect rate in the Pilot Project.

My Answer: D

I wonder what this all has to do with testing? I can name animals but I'm not a vet.

Score: 21/39

40. Below is a list of test efficiency improvement goals a software development and test organization would like to achieve. 
Which of these goals would best be supported by a test management tool? 
a) To build traceability between requirements, tests, and bugs.
b) To optimize the ability of tests to identify failures.
c) To resolve defects faster.
d) To automate selection of test cases for execution.
My Answer: D

What's a test management tool? A tool to manage testing? I do that myself. Unless you mean to manage the test project? I do that too. What does this even mean? I chose D because A is pointless, B is impossible, C is just plain false (the opposite may be true) and there are tools that can generate test cases for you based on inputs (like Hexawise).

Score: 21/40

The Result

I scored 21/40, which is 52.5%. I required 65% to pass, which would mean I'd have to have answered 26 of the 40 questions correctly.

This means that I needed to get 5 more points to have passed the exam and gotten Foundation level certification. It means that I am permitted to answer 14 of these questions incorrectly and get Foundation level certification.

I don't want to disturb anyone, but it seems to me that it's remarkably easy to get this certification provided you play ball and answer based on what you expect of them.

Final Thoughts

Well, we've all had some fun over the last few days, but there are some serious points to be made here. I got 52.5% on this exam by forgetting to answer questions, being facetious, answering honestly and not studying the course materials or syllabus. We can probably say that if I'd studied I would have passed, and so would nearly anyone else who wanted to. We can also probably say that if someone within the test industry can get such a low score it's questionable if the content of the exam.tests for useful knowledge.

I know I've berated some of the answers in this exam. I expected to. I took this, thinking it might help to clear up for me why the ISTQB is so hated by some with a little first-hand knowledge. I expected a highly academic tightly-written exam that glossed over social sciences and self-learning in exchange for repetition of terms and testing the knowledge of techniques and documentation. I found that, but what I found exceeded my own expectations of such a divide. I have enough problems with the questions in this exam to ensure no interest in the materials. Taking this test has taught me so much about the ISTQB exam already and frankly the exam paper reads like a scam. It's like renaming fruit with names of vegetables, then showing you pictures of the fruit and getting you to utter the name of the associated vegetable. Then charging you money and giving you paper in order to pass a contrived barrier to the field of testing, and everything that that implies.

Firstly, there's the quality of the questions themselves. There's no precision of language, misuse of commas, grammatical errors, terms given in confusing English, and questions that can be logically reduced to 50% guesses or solved without testing knowledge. Secondly, the content of the questions. It's well known that the ISTQB exam does not test for skill or ability. It tests for knowledge. But what knowledge, and how useful is it? Is it useful to know what equivalence partitioning is? Yes. But what about how it works, when it's used, how it's useful, when it's not useful, when it fails, in what situations it's probably more costly than valuable. What about practising on real world examples? What about defending an opinion on it, not just presenting one or being given one by the exam board? What about development of skill? None of this seems to be tested in the exam. Don't get me wrong, it may or may not be taught on the course (I've never studied for it and I've never taken it) but the exam doesn't test for it. So no matter how good the course is people who misunderstand testing will pass because of the quality of the exam questions and people who do understand testing will fail because of the quality of the exam questions.

What I found was confused, badly written, badly designed questions asking pointless things that do not test even the knowledge of a factory school course let alone an understanding of testing and how to do it. No matter what your camp, school, field, knowledge, skill or experience I'd seriously consider examining the claims of this certification versus what it actually is capable of doing. If you want to be a good tester you have to ask questions of the thing under test. Be a good tester, and ask questions of certifications. You'll come up with your own opinion, you certainly don't need me to tell you that, but you can't have an informed opinion without first informing yourself. And if you still want to take the certification, you're going to need to take the material with enough salt to risk sodium poisoning.

Let's just say this: If I show you that someone that got 100% on this exam would it positively influence your decision to hire them?

So what are your thoughts? Any insights on the questions? Anything you disagree with? Let me know in the comments, or contact me on Twitter @kinofrost.

Again, the exam and marking sheet are all under copyright owned by the ISTQB.

Good follow-ups to this post by better writers are below:

James Bach -
Alan Richardson -
Rob Lambert -
Pradeep Soundararajan -

I Failed The ISTQB Practice Exam - Part V

Series: Part IPart IIPart III & IVPart VPart VI and Final Thoughts

I was on 15/28...

Part V - Test Management

29. Which of the following best describes the task partition between test manager and tester?

a) The test manager plans testing activities and chooses the standards to be followed, while the tester chooses the tools and controls to be used. 
b) The test manager plans, organizes and controls the testing activities, while the tester specifies, automates and executes tests. 
c) The test manager plans, monitors and controls the testing activities, while the tester designs tests. 
d) The test manager plans and organizes the testing and specifies the test cases, while the tester prioritizes and executes the tests. 

My Answer: None / it depends

It depends on the company. In A sometimes testers choose tools to be used, in D sometimes the tester specifies test cases and the manager prioritizes testing.
What's really interesting go me is the answer in B (also in C) that the manager controls the testing activities. Only if you hire dead testers and tie strings to them like a macabre puppeteer. Hey testers, do you have any control over your test activities? That is to say when you're doing the things you do to test do you have any control over the things you do to test? Does your manager have any direct control over this at all?

Well you're wrong because it's B.

Score: 15/29

30. Which of the following can be categorized as product risks?

a) Low quality of requirements, design, code and tests.
b) Political problems and delays in especially complex areas in the product.
c) Error-prone areas, potential harm to the user, poor product characteristics.
d) Problems in defining the right requirements, potential failure areas in the software or system.

My Answer: C

A is project, B is project, D is project, C is product. Me good test now.

Score: 16/30

31. Which of the following are typical test exit criteria?
a) Thoroughness measures, reliability measures, test cost, schedule, state of defect correction and residual risks.
b) Thoroughness measures, reliability measures, degree of tester independence and product completeness. 
c) Thoroughness measures, reliability measures, test cost, time to market and product completeness, availability of testable code.
d) Time to market, residual defects, tester qualification, degree of tester independence, thoroughness measures and test cost.

My Answer: A

What are "test exit criteria"?  For me that's things like "end of the session" or "I'm bored now" or "Not much more I can do here" or "time for a break". Things like that. How do we measure thoroughness? Seriously, measure my thoroughness, I dare you. Then measure my reliability. I like to use a tricorder. State of defect correction is a test exit criterion?
This doesn't make any sense. I'm so confused. Oh, the answer was a guess.

Score: 17/31

32. As a Test Manager you have the following requirements to be tested: 
Requirements to test: 
R1 - Process Anomalies – High Complexity 
R2 - Remote Services – Medium Complexity 
R3 – Synchronization – Medium Complexity 
R4 – Confirmation – Medium Complexity 
R5 - Process closures – Low Complexity 
R6 – Issues – Low Complexity 
R7 - Financial Data – Low Complexity 
R8 - Diagram Data – Low Complexity 
R9 - Changes on user profile – Medium Complexity 

Requirements logical dependencies (A -> B means that B is dependent on A): 
How would you structure the test execution schedule according to the requirement dependencies? 

a) R4 > R5 > R1 > R2 > R3 > R7 > R8 > R6 > R9.
b) R1 > R2 > R3 > R4 > R5 > R7 > R8 > R6 > R9.
c) R1 > R2 > R4 > R5 > R3 > R7 > R8 > R6 > R9.
d) R1 > R2 > R3 > R7 > R8 > R4 > R5 > R6 > R9.

My Answer: I wouldn't.

Who in the name of buttery buttocks structures a test execution schedule ahead of time? If you've worked on a serious product and had a test execution schedule (a schedule for executing tests) that actually says when what tests are executed and stuck to it I'll give you 20 quid. Why are we assuming that this document is accurate? Who wrote it and why? Are these really the dependencies? According to whom? What about changes to the project? What about parallel activities or early availability of functionality or documentation? Can we express the logical relationship between explicit requirements with an arrow? Can we do it with implicit/tacit requirements at all? Hint: no. There's an infinite number of places where this schedule can fail. Is it important to schedule by dependency in whatever this example would be in real life? These are the interesting questions.
So no, I would not just write a test execution schedule, I'd sketch a plan based on things like availability, testability, deadlines, deliverables, and not just complexity but RISK.
I'd like to do an experiment. Change the question to just show the diagram, the answers, and a big question mark and see if the pass rate for this question remains the same.

Score: 17/32

33. What is the benefit of independent testing?

a) More work gets done because testers do not disturb the developers all the time.
b) Independent testers tend to be unbiased and find different defects than the developers.
c) Independent testers do not need extra education and training.
d) Independent testers reduce the bottleneck in the incident management process.

My Answer: D?

What is independent testing? Independent of what? Freedom of thought? I could recommend some books and scientific papers indicating that independent testers have a raft of biases. And what evidence is there that they find "different" defects? Different how? Do they necessarily and all the time find only different defects?This question is so infuriating.

Score: 17/33

34. Which of the following would be categorized as project risks?

a) Skill and staff shortages.
b) Poor software characteristics.
c) Failure-prone software delivered.
d) Possible reliability defect (bug).

My Answer: All of them, obviously

Okay, if you have a skill/staff shortage that could run a risk to your project, depending on what the skill and staff shortages are. I mean, we have a huge skill shortage when it comes to arc welding, architecture and railway maintenance but it's never seemed to come up.
If the software has "poor characteristics" then that could be a project risk. For example if the product has the poor characteristic of "not very testable" then testers may not be able to provide fast useful feedback to test clients which means that people are making uninformed decisions which is a project risk.
Delivering failure-prone software is a project risk. It's a risk to the success of the project. Delivering a failure-prone product is definitely a risk to the success of the project. I don't know how else to put this.
A "possible reliability defect" is a risk (as it's only a possible bug) and it threatens the success of the project so it's a project risk.

Am I just being dense? Anyway, the correct answer is A.

Score: 17/34

35. As a test manager you are asked for a test summary report. Concerning test activities and according to IEEE 829 Standard, what should you consider in your report? 

a) The number of test cases using Black Box techniques.
b) A summary of the major testing activities, events and its status in respect of meeting goals.
c) Overall evaluation of each development work item.
d) Training taken by members of the test team to support the test effort.

My Answer: B

I don't know the IEEE 829 standard. That didn't help as I wasn't allowed to look it up. I've never needed to. I've never worked in an IEEE 829 standard environment. And if I did the company would ensure I knew IEEE 829 well enough. So why is there a question on it here? Do they teach IEEE 829 for the exam? But hey, if there's anything that says "efficiency" is standardisation of test documentation, right? How about "report what's needed to the right person in the right way at the right time"? I like that as a standard. I'll call it Common Sense 829.
I guessed the answer they wanted. The actual answer is: you'd consider them all if you felt it was necessary to summarise your testing. Say you needed to train your test team to use a helpful tool in order to better test the product? I'd include that in my report to a manager who wants to know what I've been doing with my time and why more testing wasn't done.

Score: 18/35

36. You are a tester in a safety-critical software development project. During execution of a test, you find out that one of your expected results was not achieved. You write an incident report about it. What do you consider to be the most important information to include according to the IEEE Std. 829? 

a) Impact, incident description, date and time, your name.
b) Unique id for the report, special requirements needed.
c) Transmitted items, your name and you’re feeling about the defect source.
d) Incident description, environment, expected results.

My Answer: A?

Again, I don't know the standard. A is the only one that contained both description and impact. D was another candidate as it contained environment and expectation (oracle information). What  the gibbering testicle are "transmitted items" anyway? I don't know "you're" feeling on the matter.*

* Remember, this is the practice exam directly downloaded from the ISTQB's own website, provided by them as a workable example.

Score: 19/36

Next time, the thrilling conclusion and my final thoughts! See you then!

Wednesday, June 18, 2014

I Failed The ISTQB Practice Exam - Part III and IV

Series: Part IPart IIPart III & IVPart VPart VI and Final Thoughts

Welcome back! Static Techniques is short, so I've thrown in Test Design Techniques at no extra charge. I was on 7/13...

Part III - Static Techniques

14. Which of the following are the main phases of a formal review?

a) Initiation, status, preparation, review meeting, rework, follow up.
b) Planning, preparation, review meeting, rework, closure, follow up.
c) Planning, kick off, individual preparation, review meeting, rework, follow up.
d) Preparation, review meeting, rework, closure, follow up, root cause analysis.

My answer: ?
ISTQB says: C

A formal review of what? Any review of anything that conforms to a particular structure. Okay. So any of these could be the main phases of a review that is formal. The question is actually asking "which of the following are the main phases of the kind of formal review that I'm thinking of right now". I couldn't have answered this without reading the material and memorising a specific kind of formal review.

Score: 7/14

15. Which TWO of the review types below are the BEST fitted (most adequate) options to choose for reviewing safety critical components in a software project? 
Select 2 options. 

a) Informal review.
b) Management review.
c) Inspection.
d) Walkthrough.
e) Technical Review.

My answer: C & E, it depends
ISTQB says: C and E

I gave myself the marks, because I had the same answer. I don't know why these are always the most adequate options... unless they're just the most adequate out of the selection. An informal review (by my terminology as any review that is informal) can have powerful results in terms of learning. A review by management may be helpful in some circumstances - even a review of management might help. Inspection is the stupidest answer I've ever heard for anything - of course you're going to inspect the safety critical components. 1/40th of this exam rides on the answer to the question "do you think you should look at safety critical components when reviewing them?" being "yes". Walkthroughs are incredibly useful for understanding the structure, interface and purpose of components. Technical reviews are... also useful. I just picked two because the question told me to pick two, but I could debate the case for each. Oh, and what the hell is a "component" in the context of the project, and what is the project for? To replicate the feeling I'm having here imagine having George R R Martin on your radio show about improving creating writing and asking him "so, which is better, pencil or pen?".

Score: 8/15

16. Which of the following statements about static analysis is FALSE?

a) Static analysis can be used as a preventive measure with appropriate process in place. 
b) Static analysis can find defects that are not easily found by dynamic testing.
c) Static analysis can result in cost savings by finding defects early.
d) Static analysis is a good way to force failures into the software.

My answer: D
ISTQB says: D

What an utter waste of words. You don't need me to explain this to you, do you? Which of the following is FALSE about Test Technique Foo? a) It can be used to do good things, b) It can help with stuff, c) It's totally good, d) It will make your software worse by magic. Thank you, now I completely understand your knowledge of Test Technique Foo.

Score: 9/16

Part IV - Test Design Techniques

17. One of the test goals for the project is to have 100% decision coverage. The 
following three tests have been executed for the control flow graph shown 

Test A covers path: A, B, D, E, G. 
Test B covers path: A, B, D, E, F, G. 
Test C covers path: A, C, F, C, F, C, F, G. 

Which of the following statements related to the decision coverage goal is 

a) Decision D has not been tested completely.  
b) 100% decision coverage has been achieved.  
c) Decision E has not been tested completely.  
d) Decision F has not been tested completely.  

My answer: I want nothing to do with this
ISTQB says: A

Anyone that can follow a set of arrows could have answered this question, but I stuck to my principles with this one. Obviously the path A, B, D, F isn't covered by the test cases. But why should we care?
Firstly, this is a diagram and not a program. How helpful is it to have this diagram without knowing what it's modelling? What is it leaving out? What ARE these decisions? And what can we honestly and reliably say about them being "tested completely". Okay, so you've gone through your if statement and it's gone to each direction and it hasn't errored. What value does that give you? What have you learned about the system and if it's fit for use?
Secondly what on earth are these tests, and what does it mean to "cover" a path? Are the paths really covered by those tests? That's an important question. What about hidden paths? Say you're testing a web application and the session times out. Say you pull the plug out of your desktop machine. That's a path too. You can look at the decisions independently (see what conditional statements the workflow relies upon) but what does it mean to weave them all together in this graph?
I will not answer this question without my curiosity assuaged, sir. And for my impertinence I am punished by one point.

Score: 9/17

18. A defect was found during testing. When the network got disconnected while receiving data from a server, the system crashed. The defect was fixed by correcting the code that checked the network availability during data transfer. The existing test cases covered 100% of all statements of the corresponding module. To verify the fix and ensure more extensive coverage, some new tests were designed and added to the test suite. 

What types of testing are mentioned above?

A. Functional testing.
B. Structural testing. 
C. Re-testing. 
D. Performance testing. 

a) A, B and D.  
b) A and C.  
c) A, B and C.  
d) A, C and D.  

My answer: Type C, so none
ISTQB says: C

Was functional testing mentioned? Well the statement "A defect was found during testing" could be any testing. "When the network got disconnected while receiving data from a server, the system crashed" could be as part of testing I suppose, and if it is then the type of testing used wasn't mentioned.  Re-testing is mentioned, as something was tested again. Performance testing - we'll never know if someone was specifically testing for performance issues.

Score: 9/18

19. Which of the following statements about the given state table is TRUE? 

a) The state table can be used to derive both valid and invalid transitions.
b) The state table represents all possible single transitions.
c) The state table represents only some of all possible single transitions.
d) The state table represents sequential pairs of transitions.

My answer: C
ISTQB says: B

Okay. A is actually true. It could be used to derive valid and invalid transitions. It does not represent all possible single transitions that exist because this is an abstract model. Therefore C is correct. D is...

I'm wasting my time. Why is this important? How am I a better tester for this? The more I analyse what I wrote the more I question why I bothered to write it.

Score: 9/19

20. Which of the following statements are true for the equivalence partitioning test technique? 
A. Divides possible inputs into classes that have the same behaviour.
B. Uses both valid and invalid partitions.
C. Makes use only of valid partitions.
D. Must include at least two values from every equivalence partition.
E. Can be used only for testing equivalence partitions inputs from a Graphical User Interface.

a) A, B and E are true; C and D are false.
b) A, C and D are true; B and E are false.
c) A and E are true; B, C and D are false.
d) A and B are true; C, D and E are false.

My answer: D
ISTQB says: D

This sort of examines if I understand what equivalence partitioning is. Not what it's for, how it's used, when it fails, just what it is. A is basically the case, B is also basically the case. If B is true then C can't be true. D is just an arbitrary rule. E is the realm of the unhinged.

Knowing that E is probably wrong to the layman we can narrow down our choices to (b) and (d). Both contain A as true. B and C are mutually exclusive, so we don't have to know D. So with a little applied logic we can safely say that this question can be replaced with:

Does the equivalence partitioning test technique make use of both valid and invalid partitions? YES/NO.

Just trying to save you the ink, guys.

Score: 10/20

21. Which TWO of the following solutions below lists techniques that can all be categorized as Black Box design techniques?
Select 2 options. 

a) Equivalence Partitioning, decision tables, state transition, and boundary value.
b) Equivalence Partitioning, decision tables, use case.
c) Equivalence Partitioning, decision tables, checklist based, statement coverage, use case.
d) Equivalence Partitioning, cause-effect graph, checklist based, decision coverage, use case.
e) Equivalence Partitioning, cause-effect graph, checklist based, decision coverage and boundary value.

My answer: A and B
ISTQB says: A and B

Okay, cross out equivalence partitioning as it's in all of them. Decision tables split A, B and C from D and E, so the answer is either (d) and (e) or some part of (a), (b) or (c). See, we're learning testing! Now then, "checklist based" is found in (c), (d) and (e) which means that (c) cannot be correct. So it's either (a) and (b) or (d) and (e) - down to a 50% chance of guessing!

(a) and (b) contain: EP, DT, ST, BV, UC.
(d) and (e) contain: EP, CEG, CB, DC, UC, BV.

Removing common elements:
(a) and (b) contain: DT, ST
(d) and (e) contain: CEG, CB, DC

So is the answer "Decision tables and state transition" or is it "cause-effect graph, checklist based and decision coverage".

Well, the shorter list is more likely, I went for that one.

That is the entirety of the knowledge I used in answering this question. This is not an interesting or useful question.

Score: 11/21

22. An employee’s bonus is to be calculated. It cannot become negative, but it can be calculated to zero. The bonus is based on the duration of the employment. An employee can be employed for less than or equal to 2 years, more than 2 years but less than 5 years, 5 to 10 years, or longer than 10 years. Depending on this period of employment, an employee will get either no bonus or a bonus of 10%, 25% or 35%. How many equivalence partitions are needed to test the calculation of the bonus?
a) 3.
b) 5.
c) 2.
d) 4.

My answer: D
ISTQB says: D

It CANNOT BECOME NEGATIVE. The laws of nature simply preclude it from being so. God himself has written the validation - etched it into the soul of the universe. If you can count a list of items you can guess the probable "answer" to this question. Of course in reality context needed, it depends, asking questions, blah, blah.

Score: 12/22

23. Which of the following statements about the benefits of deriving test cases from use cases are most likely to be true? 
A. Deriving test cases from use cases is helpful for system and acceptance testing. 
B. Deriving test cases from use cases is helpful only for automated testing. 
C. Deriving test cases from use cases is helpful for component testing. 
D. Deriving test cases from use cases is helpful for testing the interaction between different components of the system. 

a) A and D are true; B and C are false.  
b) A is true; B, C and D are false.  
c) A and B are true; C and D are false.  
d) C is true; A, B and D are false.  

My answer: All of them, so none of the answers.
ISTQB says: A

There are situations in which all of them are most true. A can be true, B can be true, C can be true, D can be true. Depends on the test cases and the use cases and the situation. Usually use cases are flawed and partially missing with mistakes in them and not maintained for changes, so I was looking for "Deriving test cases from use cases is probably a bad idea". What does it even mean? Derive them from use cases? You mean do the use cases and see what happens? Well then that's all of them isn't it. Someone please tell me I'm wrong, I really want to know why A is correct and I'm wrong.

Thinking about it, likelihood is a very hard thing to calculate. We'd need to know a statistically significant number of unbiased selected occurrences of such derivations and when it was helpful and when it was not helpful for each.

What a bloody stupid question.

Score: 12/23

24. Which of the below would be the best basis for fault attack testing?

a) Experience, defect and failure data, knowledge about software failures.  
b) Risk analysis performed at the beginning of the project.  
c) Use Cases derived from the business flows by domain experts.  
d) Expected results from comparison with an existing system.  

My answer: A?
ISTQB says: A

I don't know what fault attack testing is and I guessed A. I don't know why it's better than risk analysis or whatever but I know faults (something that allows a product to exhibit a problem) and failures (something the product does that we wish it wouldn't) are directly linked. Threats cause faults to show failures. So I guessed A.

Score: 13/24

25. Which of the following would be the best test approach when there are poor specifications and time pressures? 

a) Use Case Testing.  
b) Condition Coverage.  
c) Exploratory Testing.  
d) Path Testing.  

My answer: C
ISTQB says: C

C is the only approach listed. Which of these vehicles has the highest top speed? a) Banana, b) Spoon, c) Motorbike, d) Soil

Score: 14/25

26. Which one of the following techniques is structure-based?

a) Decision testing.  
b) Boundary value analysis.  
c) Equivalence partitioning.  
d) State transition testing.  

My answer: D?
ISTQB says: A

What does that even mean? Structure, to me, means code and hardware and sample data and packaging and paper documents - the building blocks of the product. So decision testing - well it's not based on structure. Boundary values... again, not based on structure. None of them are, I just got desperate and put D.

Score: 14/26

27. You have started specification-based testing of a program. It calculates the greatest common divisor (GCD) of two integers (A and B) greater than zero. 

calcGCD (A, B); 

The following test cases (TC) have been specified. 

INT_MAX: largest Integer 

Which test technique has been applied in order to determine test cases 1 through 6? 

a) Boundary value analysis.
b) State transition testing.
c) Equivalence partitioning.
d) Decision table testing.

My answer: A & C
ISTQB says: A

I have started specification-based testing have I? Okay, what else dwells within this dystopian hell? Looks boundary value-y. Also to separate the nature of the inputs they've been put in equivalence classes. But I suppose it depends on what the damn specifications contain, doesn't it?

Score: 14/27

28. Consider the following state transition diagram and test case table: 

Which of the following statements are TRUE?

A. The test case table exercises the shortest number of transitions.
B. The test case gives only the valid state transitions.
C. The test case gives only the invalid state transitions.
D. The test case exercises the longest number of transitions.

a) Only A is true; B, C and D are false.
b) Only B is true; A, C and D are false.
c) A and D are true; B, C are false.
d) Only C is true; A, B and D are false.

My answer: B
ISTQB says: B

"The test case gives only the valid state transitions". I'm guessing they mean the test case table. Let me just say that I don't really fully understand the nature of the question, I just guessed this to be correct.

Score: 15/28

Anyone else as exhausted as I am? If not, meet me next time for "Test Management"!

Tuesday, June 17, 2014

I Failed The ISTQB Practice Exam - Part II

Series: Part IPart IIPart III & IVPart VPart VI and Final Thoughts

Welcome back, failure fans! For those following closely I'm currently on 5/7.

Next up...

Part II - Testing Throughout The Software Life Cycle

8. Which statement below BEST describes non-functional testing?

a) The process of testing an integrated system to verify that it meets specified requirements. 
b) The process of testing to determine the compliance of a system to coding standards. 
c) Testing without reference to the internal structure of a system.  
d) Testing system attributes, such as usability, reliability or maintainability.  

My Answer: D

There is plenty of room here for the non-functional vs quasi-functional arguments. Also D isn't the only possible answer, there's an argument for A, B and C as they are all possibly "non-functional" testing, but D just seemed like the obvious choice. A little too obvious.

Score: 6/8

9. What is important to do when working with software development models? 

a) To adapt the models to the context of project and product characteristics. 
b) To choose the waterfall model because it is the first and best proven model. 
c) To start with the V-model and then move to either iterative or incremental models. 
d) To only change the organization to fit the model and not vice versa.  

My Answer: A

Why would anyone just choose Waterfall or V-model just because without looking at it as a solution for their company? The organisation must change to fit the model, but obviously we don't "only change" it that way around. That leaves A. The most positive-sounding one there is.

Also why is this a foundation question for a tester? I test in the methodology I'm given, personally, maybe this is more variable than I expect.

Score: 7/9

10. Which of the following characteristics of good testing apply to any software development life cycle model? 

a) Acceptance testing is always the final test level to be applied.  
b) All test levels are planned and completed for each developed feature.  
c) Testers are involved as soon as the first piece of code can be executed.  
d) For every development activity there is a corresponding testing activity.  

My Answer: None

A is false (if we're applying "test levels") and is not a "characteristic of good testing", I'm not sure about B, maybe one would do such a thing in some circumstance or other, I don't think that it's important. C is false, they should probably be involved as early as possible.

D is interesting. What does "corresponding" mean? Do they mean coding, or all development? Writing the source code is a development activity - what is the corresponding testing activity? Writing test code? Yes testers can code, but if that's "corresponding" then what value does this information have? We both can drink coffee too. Here's a development activity: writing a use case document. What't the corresponding testing activity? Here's another: researching code libraries to handle file compression. What's the corresponding testing activity there? Maddening. So I say none of them are true. I am wrong, though, the answer was "D".

Score: 7/10

11. For which of the following would maintenance testing be used?

a) Correction of defects during the development phase.  
b) Planned enhancements to an existing operational system.  
c) Complaints about system quality during user acceptance testing.  
d) Integrating functions during the development of a new system.  

My Answer: C?

What on earth is "maintenance testing"? Here I feel that reading the material would have been helpful in answering the question. Correction of defects sounds like a coding endeavour (although what does it really mean?). Complaints during UAT sounds like it might be done during some sort of maintenance of the shipped product which why I guessed it. Planned enhancements - so testing the plans to add things to an existing system? Integrating functions during development - that's not even a testing activity.

But no, it's B. Okay, listen to this as the answer: "Maintenance testing would be used for planned enhancements to an existing operational system". Would it? What is it? Why would it be used? What makes it different to any other testing? If this is taught as part of the course it isn't tested in this exam.

I'll re-write this question's answers to show how effective I think it is:
For which of the following would maintenance testing be used?
a) Driving to school
b) Testing the user story cards for changes we want to make in the live environment
c) Testing calls we get during UAT
d) Making the tea

And guess what - the ISTQB site says Maintenance testing is "testing an existing system" - which means that ALL TESTING OF ANY EXISTING PRODUCT IS MAINTENANCE TESTING. (source: here, look at 2.4)

Gah! Damn you, marking sheet.

Score: 7/11

12. Which of the following statements are TRUE? 
A. Regression testing and acceptance testing are the same. 
B. Regression tests show if all defects have been resolved. 
C. Regression tests are typically well-suited for test automation. 
D. Regression tests are performed to find out if code changes have introduced or uncovered defects. 
E. Regression tests should be performed in integration testing. 

a) A, C and D and E are true; B is false.  
b) A, C and E are true; B and D are false.  
c) C and D are true; A, B and E are false.  
d) B and E are true; A, C and D are false.  

My Answer: None

Regression testing and acceptance testing are obviously different. Nothing will show if all defects have been resolved... although I question what that even means. Regression tests are typically well-suited for test automation. That doesn't even make any sense - what kind of regression tests? What kind of automation? Anyway, automation is a tool to assist all testing, and automation is often expensive and difficult, so I reason that regression tests are not typically well-suited for test automation. If you want more examples I could say that regression tests (tests to look for a backslide in quality) are not necessarily or even usually checks. I don't know about D. Is it equivalent to say "to find out if code changes have introduced or uncovered defects regression tests are performed"? If so that I could get behind D as an answer. E... well, maybe, it depends.

Score: 7/12

13. Which of the following comparisons of component testing and system testing are TRUE?

a) Component testing verifies the functioning of software modules, program objects, and classes that are separately testable, whereas system testing verifies interfaces between components and interactions with different parts of the system.
b) Test cases for component testing are usually derived from component specifications, design specifications, or data models, whereas test cases for system testing are usually derived from requirement specifications, functional specifications or use cases.
c) Component testing focuses on functional characteristics, whereas system testing focuses on functional and non-functional characteristics.
d) Component testing is the responsibility of the technical testers, whereas system testing typically is the responsibility of the users of the system.

My Answer: A

C and D are clearly bizarre. I imagine a written driving test with answers like "a) run him over, b) run over his child, c) run over his dog, d) stop at the red light".

A - Component testing verifies the functioning of (components), whereas system testing verifies (the system). Seemed okay to me, even if the wording was rather dodgy. Apparently not, though as the answer is "test cases for component testing are usually derived from component specifications, design specifications, or data models, whereas test cases for system testing are usually derived from requirement specifications, functional specifications or use cases". I don't really know what that means. Are they explicit or tacit? Are they specifications and models or specification and model documents? What about the other oracles for testing? Why would the model we build for testing be reliant on only one set of specifications? Aren't test cases derived from tacit contextual information as much as explicit specification? Otherwise how would you know the value of your testing? And what about test cases derived from testing, surely they're more common?

Just me being over-critical in my thinking, because it's definitely B on the score sheet.

Score: 7/13

That's it, exam fans! Meet up with me next time for "Static Techniques"!

Monday, June 16, 2014

I Failed The ISTQB Practice Exam - Part I

Series: Part I, Part II, Part III & IV, Part V, Part VI and Final Thoughts

Oh, the ISTQB! A source of confusion and marketing evil that the devil himself could learn from. Or maybe it's the one standard of the beginners of our industry that sets some common domain language bringing everyone together in a way that will eventually unify the world under a flag of peace.

Either way I wanted to take the ISTQB practice paper and see how I'd do if someone just thrust it in front of me. So here were the rules:

1. No looking anything up.
2. No reading of the ISTQB materials, syllabus, website, etc.
3. Try to answer honestly
4. Use this practice paper: (ISTQB CTFL Sample Exam Paper), marked using the marking sheet: (Answer Sheet for Practice Exam Version 2011)
5. If the ISTQB's marking sheet says I don't get marks then I don't get marks.

Note: The ISTQB is the source and copyright owner of this practice exam. I republish any materials therein under this statement from the exam: "Any individual or group of individuals may use this Practice Exam as the basis for articles, books, or other derivative writings if ISTQB is acknowledged as the source and copyright owner of the practice exam.

The first thing I notice is that the practice exam is 4 years old, but that's what I got from their website.

So how did I do? Well I failed. The pass rate is 26/40 and I got 21/40. Oops. Let's take a look at my answers!

Part 1- Fundamentals Of Testing

1. Which of the following statements BEST describes one of the seven key 
principles of software testing? 

a) Automated tests are better than manual tests for avoiding the Exhaustive Testing. 
b) Exhaustive testing is, with sufficient effort and tool support, feasible for all software. 
c) It is normally impossible to test all input / output combinations for a software system. 
d) The purpose of testing is to demonstrate the absence of defects.  

My answer: C
ISTQB says: C

This seems fairly simple to me. I don't know what "the Exhaustive Testing" is, though or why it might want to be avoided.

Edit: Should point out that I also don't know what the "seven key principles of software testing" are. This is not required information to answer this question.

Score: 1/1

2. Which of the following statements is the MOST valid goal for a test team?

a) Determine whether enough component testing was executed.  
b) Cause as many failures as possible so that faults can be identified and 
c) Prove that all faults are identified.  
d) Prove that any remaining faults will not cause any failures. 

My answer: B
ISTQB says: B

I feel guilty about this answer. I put it because I felt it was the most valid goal for most situations out of the ones provided, but honestly I wonder now if that is the answer. B is certainly not a particularly valid goal in any case as it doesn't specify what failures are being caused. As many failures as possible, or as many useful and informative failures? I mean I could create an almost identical failure every 5 minutes and report it every 5 minutes and achieve this goal while I play Angry Birds and sip a can of Pepsi.

Score: A guilty 2/2

3. Which of these tasks would you expect to perform during Test Analysis and 

a) Setting or defining test objectives.  
b) Reviewing the test basis.  
c) Creating test suites from test procedures.  
d) Analyzing lessons learned for process improvement.  
My answer: I forgot to answer this question
ISTQB says: B

I skipped over this question and forgot to come back to it. Would I expect to set or define test objectives during test analysis and design? What is test analysis and design? Presumably they mean analysis of the situation to design tests. Well I'd probably want to define the objectives of a test I was designing (A). I don't know what test basis means - I think it's the thing that defines what the tests are going to be so we're talking about the context and the product and things I find while I'm testing so I will definitely do that (B). I might create test suites from test procedures, although that sounds very unlikely.

Score: 2/3
4. Below is a list of problems that can be observed during testing or operation. 
Which is MOST likely a failure? 

a) The product crashed when the user selected an option in a dialog box.  
b) One source code file included in the build was the wrong version.  
c) The computation algorithm used the wrong input variables.  
d) The developer misinterpreted the requirement for the algorithm.  
My answer: A
ISTQB says: A

I got this question wrong. ISTQB says I didn't, but again this is a guilty answer. They're all failures. A is a failure of the product. B is a failure of whoever put the file there. C is a failure of whoever input those variables or permitted them to be input. D is a failure of developer to interpret the requirement or of the person telling them about the requirement to make themselves understood (unless they also misinterpreted the requirement of course). I presume we're talking about an explicit requirement.

Score: A guiltier 3/4

5. Which of the following, if observed in reviews and tests, would lead to problems (or conflict) within teams? 

a) Testers and reviewers are not curious enough to find defects.  
b) Testers and reviewers are not qualified enough to find failures and faults. 
c) Testers and reviewers communicate defects as criticism against persons and not against the software product. 
d) Testers and reviewers expect that defects in the software product have already been found and fixed by the developers. 
My answer: A, C, D
ISTQB says: C

Okay, I misread the question. I actually answered the question "Which... could lead to problems...?" If testers aren't curious enough to find defects then test clients will be annoyed at the lack of service. If they communicate defects as criticism against, say, the developers this could cause a problem (but if they blamed the government perhaps not). If testers expect that defects have been found and fixed then what are they doing there? I imagine this could cause all sorts of misunderstandings.

The sentence "Testers and reviewers are not qualified enough to find failures and faults." makes absolutely no sense.

The question actually specifies that they would lead to problems, so the correct answer (in my eyes) is none of them would necessarily lead to problems.

Score: 3/5

6. Which of the following statements are TRUE? 
A. Software testing may be required to meet legal or contractual requirements. 
B. Software testing is mainly needed to improve the quality of the developer’s work. 
C. Rigorous testing and fixing of defects found can help reduce the risk of problems occurring in an operational environment. 
D. Rigorous testing is sometimes used to prove that all failures have been found. 

a) B and C are true; A and D are false.  
b) A and D are true; B and C are false.  
c) A and C are true, B and D are false.  
d) C and D are true, A and B are false.  
My answer: C (a and c are true, b and d are false)
ISTQB says: C

Software testing is obviously sometimes required for legal or contractual requirements. That's like saying "sometimes you need a buy a ticket to ride the bus". It's also obvious that finding and fixing defects can reduce risk of them occurring later on. Obviously the main point isn't to improve the developer's work.

There is a possible argument to be made for statement D being true. In a simple enough system with strictly defined parameters and with a simple enough definition of failure (say, subjective to a single user) one could define a scenario where testing proves all failures have been found. Interesting thought experiment... probably we'd have to define failures here as "product failures".

I'm fairly sure my mum could answer this question with minimum prior testing knowledge. I might try the next series as "My Mum Passes/Fails The ISTQB Practice Exam", to compare results.

Score: 4/6

7. Which of the following statements BEST describes the difference between testing and debugging?
a) Testing pinpoints (identifies the source of) the defects. Debugging analyzes the faults and proposes prevention activities. 
b) Dynamic testing shows failures caused by defects. Debugging finds, analyzes, and removes the causes of failures in the software.
c) Testing removes faults. Debugging identifies the causes of failures.
d) Dynamic testing prevents causes of failures. Debugging removes the failures.

My answer: B
ISTQB says: B

I don't think any of these answers describe the difference between testing and debugging. I just guessed what sounded most sensible. Obviously testing is so much more than showing failures caused by defects, but the first half of the statement is true. The second half isn't necessarily true. Arguments could be made for A (sometimes testing finds the source of a defect, sometimes debugging reveals prevention activities).

Score: 5/7

Well that's it so far. Come back next time for Part II - Testing Throughout The Software Lifecycle.