Test Metrics - Dangers In Communication

At the Software Testing Club meet last night many interesting points were made but one that captured my interest in particular was testing metrics.

What Are Metrics?

Testing, like most things, is full of numbers and statistics. Numbers and statistics are everywhere. And metrics are just numbers that have reason to them. Metrics are "an analytical measurement intended to quantify the state of a system"¹. 


Unfortunately numbers say whatever you would like them to say. I always return to an old adage taught to me at school:

Information = Data + Meaning

So the numbers in statistics are data, and what they refer to is the meaning. Or the numbers are the quantification and what they refer to is the state of the system. But this is all that these numbers refer to, and that is the inherent problem with metrics.

Why Do We Use Metrics?

Well to "quantify the state of the system" of course! But is that really why we use metrics? We know that's what they refer to but metrics have a habit of miscommunication meaning and I think the operative word we hit on last night was context.


Managers love metrics. They like reports, charts and graphs because they can gain information about a system that they know perhaps very little about or don't have time to understand. It's easier to go to your boss with "Our bug count is 17% lower this quarter" than "well, looks like it's going okay and the testing guys say that the quality looks higher". But do the numbers tell the truth about the system? If making you or your team look good, or putting your head down and getting away with nominal improvement is your goal then metrics are a gift from heaven.... but if relaying the true quality of your product is the goal then qualification and context are vital.


Our discussion last night prompted the example of "severity". Saying "We have 20% fewer Severity One bugs" sounds great, but this statement should prompt a lot of questions, such as:

• What on earth is a "Severity One" bug?
• What does it mean to 'have' the bug? That we found fewer? That we fixed more than we found?
• Is 'having' fewer of these bugs really a good thing? Are we spending time fixing bugs and not finding them? Are we simply not finding them any more when we should?

Science!

Science is a methodology for finding truth about the universe despite our own cognitive biases. One look at an optical illusion shows us that seeing isn't believing, and starting with an answer and then trying to prove it makes the likelihood of experimental data showing the pre-selected answer more likely ("hmmn, that doesn't look right, I'll take that reading again..."). We are pattern seeking animals, we tend to confirm our own beliefs and look for "hits" and forget the "misses" - this is confirmation bias. It's why we see shapes in the clouds and the face of Jesus on a crisp, we are evolutionarily programmed to respond to patterns and false negatives (is that rustling in the trees wind or an enemy? Better run just in case).


All this means that we should not interpret the data after deciding what we want it to say, otherwise we are creating data that does not tell the truth. They are the most basic rules of scientific philosophy. It might make us look good or cover up our mistakes or those of our colleagues, but it's not the genuine state of the world, or the system under test.

How Should We Use Metrics?

Interpret the numbers on behalf of those receiving them, do not let them do it themselves. Those collecting the metrics are probably at the best point of understanding, so add the context and qualifications there before handing them off to someone else. Do not use confusing short-hand terms like "code coverage" and "bug count" without explanation. Formalise the meaning of domain terms like "severity" so that testers can categorise issues by a domain-wide understanding of the term.

This all feeds into our need to understand more complex things about the system than "number of bugs fixed" or "test coverage", to the ability to understand opportunity cost and apply it to schedules, risk assessment, functional prioritisation and so much more. 

 ¹ http://en.wikipedia.org/wiki/Metric

Robotics ho!

To the land of robotics!

I am going to build my first Arduino line-following robot. It is decided.

I have a shopping list, I have the tools, I may be moving house within 2 weeks so I can't have anything delivered yet, but I will start the planning process and decide where I can get a cheap chassis and some cheap wheels from... it may be time to get back to the boot sale! I can't wait to stop being ill so I can chase these things.

If you're following along at home I will need:

* LEDs
* A battery
* A chassis
* 2 hobby servo motors (which I will adapt to run continuously. I have to learn sometime.)
* Nuts/bolts
* Resistors
* Matrix board
* IR Emitter/detectors
* A motor shield
* A caster wheel for the front

I'll try to update with photos when I actually build this thing.

Organising My Life 2

Well I've tried Astrid and Toodledo and Lifetick and a host of other motivational goal-based GTD to-do list widgets and I've come to no satisfactory conclusion on them. I have found out as much what I want as what I didn't want, and I've decided what I want is three things:

1. A to-do list that is simple, has multiple lists, and contains actionable short-term things that need doing.

2. A system for maintaining personal and work projects

3. Something to hold user stories

Number one is still under consideration but at the moment I am using Wunderlist. It is simple and it works.

Number two I stumbled upon via Wunderlist, their new Wunderkit which (even without the social aspect) is a great way of braindumping my larger-scale personal projects. It holds a few points over the dishevelled notes I was holding in OneNote and Google Docs, and has a much more personable layout than both, as well as Evernote which I can't really abide.

Number three, without a doubt, is PivotalTracker which is nothing short of an amazing piece of programming, but I'm always up for suggestions!

My trouble will now be checking all these things each day! When I get home in the evening I will often not remember to check to-do lists. Wunderkit has tasks, but they don't interrupt me enough to ensure they'll get done.

So:

If I have something that needs doing I will put it on Wunderlist. The downsides are no alarms and a slightly fiddly Android app.

If I have a long-term goal or project with subtasks or recurring tasks I will use Wunderkit. The downsides are no alarms, and no visible progress towards the goal, but I like the idea and if I had other people working on a project it does make for a good project collaboration space (although I haven't needed to try others).

And I will continue to use PivotalTracker :).

What Cucumber Does

Right. Okay. Let's say I bought a toaster, then a week later tried to return it because when I dried my socks in it they caught fire. You would first question my methods of sock drying, then expand your derision to why I took it upon myself to complain that something not designed for

Cucumber is a BDD tool. It is not a testing tool, or a specification tool, or anything else. All it does is parse out human-readable behavioural specifications so that you can generate automated tests, then code software to meet those tests. Based on the quality of the specifications, AND THE INTERPRETATION LAYER OF THE CODE YOU WRITE you will get software that meets those specifications.

It is not magic. It will not make your software better. It is a tool to guide development to build software with behavioural specifications in mind, especially in an environment where there is little technical documentation (agile enviroments) or where people don't read the technical documentation more than once (everywhere else).

Yes, okay, the layer that the development team write will actually add the meaning to the top layer, but that level of code design is part of their job in the first place. Cucumber adds nothing that wasn't already there, but gives us a way of thinking about software solutions, and a framework to guide development with behaviour in mind.

Cucumber does not replace QA. It isn't really even anything to do with QA or testing.

Right. Now I have that out of my brain. Good.

Well my first jaunt into RFID taught me something. RFID reading on the Arduino is so remarkably simple that it offers about as much challange as drinking a glass of milk. It did teach me a fair amount about serial communication though, and I suppose I can't really complain about the implementation of a project being too easy. But it is. As a project it's been a complete success, but it's failed as a hobby.... So now I need to do something else with it. I've wired it into the LCD screen to show the card numbers, I wonder if I could build a game with it?

The robot arm project will also soon take off. The arm is built, but I need to Frizting up a schematic before I go blundering in stripping out the wire headers and installing h-bridge ICs.

Now sleep.

Serial Communication

Onward with the projects! I have a few things on the go at the moment, working 9 to 5:30 breaks up my creative flow for outside projects, but I've gotten the Arduino and my computer conversing nicely using serial communication. First I used a potentiometer to control the screen background in Processing (thanks to Jeremy Blum for his fantastic arduino video tutorial series, I recommend it to absolutely anyone and everyone). Next came using Processing to control the Arduino, first with an LED on/off button approach, next using sliders to control PWM brightness. I used the controlP5 sliders, you can download the controlP5 library for free (google it).

Here are the sliders on my computer screen:

And here's the results as laid out on my new breadboard (from proto-pic, where I bought the original Arduino kit):

And a blurry dark photo to emphasise the different light levels:

I could not have done this without the help of Jeremy Blum's tutorials, Lucky Larry's tutorial on Arduino/Processing serial communication, and the ControlP5 slider example by Andreas Schlegel. The LEDs are on PWM-enabled pins 10 and 11.

I ended up with this Ardunio code:

int redPin = 10;
int yelPin = 11;
int buff[2];
//int
void setup()
{
  pinMode(redPin, OUTPUT);
  pinMode(yelPin, OUTPUT); 
  Serial.begin(9600);
}

void loop()
{
  if(Serial.available() >= 2 ){
    buff[0] = Serial.read();
    buff[1] = Serial.read();
    Serial.flush();
    analogWrite(redPin, buff[0]);
    analogWrite(yelPin, buff[1]);
  }
}


And this processing code:
import processing.serial.*;
import controlP5.*;
ControlP5 controlP5;
Serial port;

int redval = 0;
int yelval = 0;

void setup() {
  size(400,400);
  smooth();
  port = new Serial(this, "COM3", 9600);
  controlP5 = new ControlP5(this);
  controlP5.addSlider("red",0,255,128,200,180,100,10);
  controlP5.addSlider("yellow",0,255,128,200,210,100,10);
}
 
void draw() {
  background(0);
}

void red(float redi) {
  redval = round(redi);
  port.write(redval);
  port.write(yelval);
}

void yellow(float yeli) {
  yelval = round(yeli);
  port.write(redval);
  port.write(yelval);
}

Which, I'll be honest, was a lot easier and shorter than I thought it was going to be. There's probably a better way of doing this (leave a comment if you know one!) but I'm happy with it for a first try. Learning with Arduino always seems to be fast, and I'm constantly surprised how simple things are. The electronics have actually been the harder part of this experience, trying to catch up after 15 or so years.

Still, having fun!

Arduino-programming girlfriend

My girlfriend picked up my Arduino, pre-wired into a 16x2 character LCD screen, and looking at a simple example has programmed it to display messages, flash text, use functions and loops, all with no previous programming experience. Jealous? Yeah, you're totally jealous. I think it also helped her understand a little of what I do at work (I'm essentially a programmer).

Personally I have a few ideas for some projects. I'd like to do some previously buillt projects first though, maybe starting with a knock sensor, then moving on to a controls-to-usb-controller-output macro thingy. i lost my last controller so I'm going to have a good time re-mapping the key matrix. At some point I have to face my fear of motors and continuous rotation servos so I can get on my way to my LOGO robot! I wonder what the torque is like on my mini servo...?

My second order from proto-pic is still awaiting fulfillment. If they're ordering in a 22mF capacitor to complete my order because I didn't check that they were out I will kick myself extra hard. My purchase from hobbytronics was cheap, very affordable postage, arrived the next day, neatly packaged, and it fit through my letterbox, so very happy with them!

Edit: Oh yes, my office is now in full force, with a workbench and a lamp for arduinoy goodness, although my notebook makes arduino work pretty mobile, and even more so when I get my components box!