Thursday, June 19, 2014

I Failed The ISTQB Practice Exam - Part VI & Final Thoughts

Series: Part IPart IIPart III & IVPart V

I was on 19/36...

Part VI - Tool Support For Testing

37. From the list below, select the recommended principles for introducing a chosen test tool in an organization? 

1. Roll the tool out to the entire organization at the same time. 
2. Start with a pilot project. 
3. Adapt and improve processes to fit the use of the tool. 
4. Provide training and coaching for new users. 
5. Let each team decide their own standard ways of using the tool. 
6. Monitor that costs do not exceed initial acquisition cost. 
7. Gather lessons learned from all teams. 

a) 1, 2, 3, 5.
b) 1, 4, 6, 7.
c) 2, 3, 4, 7.
d) 3, 4, 5, 6.

My Answer: B - only one without 3, except 6!

Obviously 3 is stupid, so (b) must be the answer! Except that it also has 6, which is rediculous. 1 would vary depending on the tool (does the CEO need RapidReporter?), 2 might be important if it's a big tool, 3 means being shaped by our tools which is dangerous, 4 might be necessary if they haven't used the tool before and it's complex or we want to use it in a certain way, 5 is a possibility (I don't teach anyone how to use Excel or a log viewer in a specific way, for example), 6 seems weird and 7 seems like a good idea. I suppose if it wasn't for 3 I might have picked C.

And who's the guy recommending these principles? Come on, someone must have had to. Unless they're sent down from on high etched into stone tablets or something.

Score: 19/37

38. Which one of the following best describes a characteristic of a keyword-driven test execution tool?

a) A table with test input data, action words, and expected results, controls execution of the system under test
b) Actions of testers recorded in a script that is rerun several times.
c) Actions of testers recorded in a script that is run with several sets of test input data
d) The ability to log test results and compare them against the expected results, stored in a text file.

My Answer: A

Another point for yours truly as he deftly answers the challenge of keyword-driven test execution by choosing the only answer that contains any test execution.

Score: 20/38

39. Which of the following is NOT a goal of a Pilot Project for tool evaluation?

a) To evaluate how the tool fits with existing processes and practices.
b) To determine use, management, storage, and maintenance of the tool and test assets.
c) To assess whether the benefits will be achieved at reasonable cost.
d) To reduce the defect rate in the Pilot Project.

My Answer: D

I wonder what this all has to do with testing? I can name animals but I'm not a vet.

Score: 21/39

40. Below is a list of test efficiency improvement goals a software development and test organization would like to achieve. 
Which of these goals would best be supported by a test management tool? 
a) To build traceability between requirements, tests, and bugs.
b) To optimize the ability of tests to identify failures.
c) To resolve defects faster.
d) To automate selection of test cases for execution.
My Answer: D

What's a test management tool? A tool to manage testing? I do that myself. Unless you mean to manage the test project? I do that too. What does this even mean? I chose D because A is pointless, B is impossible, C is just plain false (the opposite may be true) and there are tools that can generate test cases for you based on inputs (like Hexawise).

Score: 21/40

The Result

I scored 21/40, which is 52.5%. I required 65% to pass, which would mean I'd have to have answered 26 of the 40 questions correctly.

This means that I needed to get 5 more points to have passed the exam and gotten Foundation level certification. It means that I am permitted to answer 14 of these questions incorrectly and get Foundation level certification.

I don't want to disturb anyone, but it seems to me that it's remarkably easy to get this certification provided you play ball and answer based on what you expect of them.

Final Thoughts

Well, we've all had some fun over the last few days, but there are some serious points to be made here. I got 52.5% on this exam by forgetting to answer questions, being facetious, answering honestly and not studying the course materials or syllabus. We can probably say that if I'd studied I would have passed, and so would nearly anyone else who wanted to. We can also probably say that if someone within the test industry can get such a low score it's questionable if the content of the exam.tests for useful knowledge.

I know I've berated some of the answers in this exam. I expected to. I took this, thinking it might help to clear up for me why the ISTQB is so hated by some with a little first-hand knowledge. I expected a highly academic tightly-written exam that glossed over social sciences and self-learning in exchange for repetition of terms and testing the knowledge of techniques and documentation. I found that, but what I found exceeded my own expectations of such a divide. I have enough problems with the questions in this exam to ensure no interest in the materials. Taking this test has taught me so much about the ISTQB exam already and frankly the exam paper reads like a scam. It's like renaming fruit with names of vegetables, then showing you pictures of the fruit and getting you to utter the name of the associated vegetable. Then charging you money and giving you paper in order to pass a contrived barrier to the field of testing, and everything that that implies.

Firstly, there's the quality of the questions themselves. There's no precision of language, misuse of commas, grammatical errors, terms given in confusing English, and questions that can be logically reduced to 50% guesses or solved without testing knowledge. Secondly, the content of the questions. It's well known that the ISTQB exam does not test for skill or ability. It tests for knowledge. But what knowledge, and how useful is it? Is it useful to know what equivalence partitioning is? Yes. But what about how it works, when it's used, how it's useful, when it's not useful, when it fails, in what situations it's probably more costly than valuable. What about practising on real world examples? What about defending an opinion on it, not just presenting one or being given one by the exam board? What about development of skill? None of this seems to be tested in the exam. Don't get me wrong, it may or may not be taught on the course (I've never studied for it and I've never taken it) but the exam doesn't test for it. So no matter how good the course is people who misunderstand testing will pass because of the quality of the exam questions and people who do understand testing will fail because of the quality of the exam questions.

What I found was confused, badly written, badly designed questions asking pointless things that do not test even the knowledge of a factory school course let alone an understanding of testing and how to do it. No matter what your camp, school, field, knowledge, skill or experience I'd seriously consider examining the claims of this certification versus what it actually is capable of doing. If you want to be a good tester you have to ask questions of the thing under test. Be a good tester, and ask questions of certifications. You'll come up with your own opinion, you certainly don't need me to tell you that, but you can't have an informed opinion without first informing yourself. And if you still want to take the certification, you're going to need to take the material with enough salt to risk sodium poisoning.

Let's just say this: If I show you that someone that got 100% on this exam would it positively influence your decision to hire them?

So what are your thoughts? Any insights on the questions? Anything you disagree with? Let me know in the comments, or contact me on Twitter @kinofrost.

Again, the exam and marking sheet are all under copyright owned by the ISTQB.

Good follow-ups to this post by better writers are below:

James Bach -
Alan Richardson -
Rob Lambert -
Pradeep Soundararajan -

No comments:

Post a Comment